2024-06-18T02:17:43.693 INFO:root:teuthology version: 0.0.1.dev325+g8e97141 2024-06-18T02:17:43.693 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_teuthology@teuthology --verbose --archive /home/teuthworker/archive/teuthology-2024-06-17_21:08:03-orch-squid-distro-default-smithi/7760435 --name teuthology-2024-06-17_21:08:03-orch-squid-distro-default-smithi --description orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} -- /home/teuthworker/archive/teuthology-2024-06-17_21:08:03-orch-squid-distro-default-smithi/7760435/orig.config.yaml 2024-06-18T02:17:43.745 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-06-18T02:17:43.835 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-06-17_21:08:03-orch-squid-distro-default-smithi/7760435 branch: squid description: orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7760435' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/quincy ' name: teuthology-2024-06-17_21:08:03-orch-squid-distro-default-smithi no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - OSD_DOWN - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - Degraded data redundancy sha1: 3781e3a592387d16e8020f5db99cdee4198f6d00 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 3781e3a592387d16e8020f5db99cdee4198f6d00 kclient: syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: f18f3f1cbd11e4a41c557e616d172f24deb91473 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 4712 sha1: 3781e3a592387d16e8020f5db99cdee4198f6d00 sleep_before_teardown: 0 subset: 61/64 suite: orch suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_f18f3f1cbd11e4a41c557e616d172f24deb91473/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: f18f3f1cbd11e4a41c557e616d172f24deb91473 targets: smithi001.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNzD4GxedZp9rUkjdMDu8cOcEQjT8biHrjzHS/GKZ3D0BI6WXR9LG00e/Wx6uV3Ijf+MBhl+ic+v+571JgdOqMI= smithi028.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMbIu0j7dkgj8aw70x0w4FwcpuV6G+4C4GrlnI4p+kfGlmavz4+wvzQq7d6KJUqid3HaW1qHFoJdLfVmtfZLfvk= tasks: - install: branch: quincy exclude_packages: - ceph-volume - print: '**** done install task...' - cephadm: cephadm_branch: quincy cephadm_git_url: https://github.com/ceph/ceph conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.ceph.io/ceph-ci/ceph:quincy roleless: true - print: '**** done end installing quincy cephadm ...' - cephadm.shell: host.a: - ceph config set mgr mgr/cephadm/use_repo_digest true --force - print: '**** done cephadm.shell ceph config set mgr...' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 1 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay true - cephadm.shell: host.a: - ceph fs set cephfs inline_data true --yes-i-really-really-mean-it - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - ceph-fuse: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] variables: fail_fs: true teuthology_branch: main teuthology_sha1: 8e9714173de9e92c97e8ef1045d333e96b793454 timestamp: 2024-06-17_21:08:03 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph versions | jq -e '.mgr | length == 1' - ceph versions | jq -e '.mgr | keys' | grep $sha1 - ceph versions | jq -e '.overall | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 2' - ceph orch ps - cephadm.shell: env: - sha1 host.a: - ceph config set mgr mgr/orchestrator/fail_fs true - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2698841 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-06-18T02:17:43.835 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_f18f3f1cbd11e4a41c557e616d172f24deb91473/qa; will attempt to use it 2024-06-18T02:17:43.836 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_f18f3f1cbd11e4a41c557e616d172f24deb91473/qa/tasks 2024-06-18T02:17:43.836 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-06-18T02:17:43.837 INFO:teuthology.task.internal:Checking packages... 2024-06-18T02:17:43.857 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '3781e3a592387d16e8020f5db99cdee4198f6d00' 2024-06-18T02:17:43.857 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-18T02:17:43.857 INFO:teuthology.packaging:ref: None 2024-06-18T02:17:43.857 INFO:teuthology.packaging:tag: None 2024-06-18T02:17:43.857 INFO:teuthology.packaging:branch: squid 2024-06-18T02:17:43.857 INFO:teuthology.packaging:sha1: 3781e3a592387d16e8020f5db99cdee4198f6d00 2024-06-18T02:17:43.857 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-06-18T02:17:44.071 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3145.gf18f3f1c 2024-06-18T02:17:44.073 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-06-18T02:17:44.081 INFO:teuthology.task.internal:no buildpackages task found 2024-06-18T02:17:44.081 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-06-18T02:17:44.093 INFO:teuthology.task.internal:Saving configuration 2024-06-18T02:17:44.112 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-06-18T02:17:44.118 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-06-18T02:17:44.142 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi001.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-06-17_21:08:03-orch-squid-distro-default-smithi/7760435', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-06-18 02:12:58.291649', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNzD4GxedZp9rUkjdMDu8cOcEQjT8biHrjzHS/GKZ3D0BI6WXR9LG00e/Wx6uV3Ijf+MBhl+ic+v+571JgdOqMI='} 2024-06-18T02:17:44.160 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi028.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-06-17_21:08:03-orch-squid-distro-default-smithi/7760435', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-06-18 02:12:58.290302', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMbIu0j7dkgj8aw70x0w4FwcpuV6G+4C4GrlnI4p+kfGlmavz4+wvzQq7d6KJUqid3HaW1qHFoJdLfVmtfZLfvk='} 2024-06-18T02:17:44.161 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-06-18T02:17:44.167 INFO:teuthology.task.internal:roles: ubuntu@smithi001.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-06-18T02:17:44.167 INFO:teuthology.task.internal:roles: ubuntu@smithi028.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-06-18T02:17:44.167 INFO:teuthology.run_tasks:Running task console_log... 2024-06-18T02:17:44.223 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fcab347fac0>, signals=[15]) 2024-06-18T02:17:44.223 INFO:teuthology.run_tasks:Running task internal.connect... 2024-06-18T02:17:44.230 INFO:teuthology.task.internal:Opening connections... 2024-06-18T02:17:44.230 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi001.front.sepia.ceph.com 2024-06-18T02:17:44.232 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-18T02:17:44.321 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi028.front.sepia.ceph.com 2024-06-18T02:17:44.322 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-18T02:17:44.397 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-06-18T02:17:44.403 DEBUG:teuthology.orchestra.run.smithi001:> uname -m 2024-06-18T02:17:44.421 INFO:teuthology.orchestra.run.smithi001.stdout:x86_64 2024-06-18T02:17:44.422 DEBUG:teuthology.orchestra.run.smithi001:> cat /etc/os-release 2024-06-18T02:17:44.480 INFO:teuthology.orchestra.run.smithi001.stdout:NAME="CentOS Stream" 2024-06-18T02:17:44.480 INFO:teuthology.orchestra.run.smithi001.stdout:VERSION="9" 2024-06-18T02:17:44.480 INFO:teuthology.orchestra.run.smithi001.stdout:ID="centos" 2024-06-18T02:17:44.480 INFO:teuthology.orchestra.run.smithi001.stdout:ID_LIKE="rhel fedora" 2024-06-18T02:17:44.480 INFO:teuthology.orchestra.run.smithi001.stdout:VERSION_ID="9" 2024-06-18T02:17:44.480 INFO:teuthology.orchestra.run.smithi001.stdout:PLATFORM_ID="platform:el9" 2024-06-18T02:17:44.480 INFO:teuthology.orchestra.run.smithi001.stdout:PRETTY_NAME="CentOS Stream 9" 2024-06-18T02:17:44.480 INFO:teuthology.orchestra.run.smithi001.stdout:ANSI_COLOR="0;31" 2024-06-18T02:17:44.480 INFO:teuthology.orchestra.run.smithi001.stdout:LOGO="fedora-logo-icon" 2024-06-18T02:17:44.480 INFO:teuthology.orchestra.run.smithi001.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-06-18T02:17:44.480 INFO:teuthology.orchestra.run.smithi001.stdout:HOME_URL="https://centos.org/" 2024-06-18T02:17:44.481 INFO:teuthology.orchestra.run.smithi001.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-06-18T02:17:44.481 INFO:teuthology.orchestra.run.smithi001.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-06-18T02:17:44.481 INFO:teuthology.orchestra.run.smithi001.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-06-18T02:17:44.481 INFO:teuthology.lock.ops:Updating smithi001.front.sepia.ceph.com on lock server 2024-06-18T02:17:44.504 DEBUG:teuthology.orchestra.run.smithi028:> uname -m 2024-06-18T02:17:44.523 INFO:teuthology.orchestra.run.smithi028.stdout:x86_64 2024-06-18T02:17:44.523 DEBUG:teuthology.orchestra.run.smithi028:> cat /etc/os-release 2024-06-18T02:17:44.581 INFO:teuthology.orchestra.run.smithi028.stdout:NAME="CentOS Stream" 2024-06-18T02:17:44.581 INFO:teuthology.orchestra.run.smithi028.stdout:VERSION="9" 2024-06-18T02:17:44.582 INFO:teuthology.orchestra.run.smithi028.stdout:ID="centos" 2024-06-18T02:17:44.582 INFO:teuthology.orchestra.run.smithi028.stdout:ID_LIKE="rhel fedora" 2024-06-18T02:17:44.582 INFO:teuthology.orchestra.run.smithi028.stdout:VERSION_ID="9" 2024-06-18T02:17:44.582 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM_ID="platform:el9" 2024-06-18T02:17:44.582 INFO:teuthology.orchestra.run.smithi028.stdout:PRETTY_NAME="CentOS Stream 9" 2024-06-18T02:17:44.582 INFO:teuthology.orchestra.run.smithi028.stdout:ANSI_COLOR="0;31" 2024-06-18T02:17:44.582 INFO:teuthology.orchestra.run.smithi028.stdout:LOGO="fedora-logo-icon" 2024-06-18T02:17:44.582 INFO:teuthology.orchestra.run.smithi028.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-06-18T02:17:44.582 INFO:teuthology.orchestra.run.smithi028.stdout:HOME_URL="https://centos.org/" 2024-06-18T02:17:44.582 INFO:teuthology.orchestra.run.smithi028.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-06-18T02:17:44.582 INFO:teuthology.orchestra.run.smithi028.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-06-18T02:17:44.582 INFO:teuthology.orchestra.run.smithi028.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-06-18T02:17:44.583 INFO:teuthology.lock.ops:Updating smithi028.front.sepia.ceph.com on lock server 2024-06-18T02:17:44.604 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-06-18T02:17:44.612 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-06-18T02:17:44.616 INFO:teuthology.task.internal:Checking for old test directory... 2024-06-18T02:17:44.617 DEBUG:teuthology.orchestra.run.smithi001:> test '!' -e /home/ubuntu/cephtest 2024-06-18T02:17:44.619 DEBUG:teuthology.orchestra.run.smithi028:> test '!' -e /home/ubuntu/cephtest 2024-06-18T02:17:44.638 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-06-18T02:17:44.644 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-06-18T02:17:44.644 DEBUG:teuthology.orchestra.run.smithi001:> test -z $(ls -A /var/lib/ceph) 2024-06-18T02:17:44.676 DEBUG:teuthology.orchestra.run.smithi028:> test -z $(ls -A /var/lib/ceph) 2024-06-18T02:17:44.701 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-06-18T02:17:44.756 INFO:teuthology.run_tasks:Running task kernel... 2024-06-18T02:17:44.769 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-06-18T02:17:44.769 INFO:teuthology.task.kernel:config {'host.a': {'kdb': 1, 'sha1': 'distro'}, 'host.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-06-18T02:17:44.770 DEBUG:teuthology.orchestra.run.smithi001:> test -f /run/.containerenv -o -f /.dockerenv 2024-06-18T02:17:44.770 DEBUG:teuthology.orchestra.run.smithi028:> test -f /run/.containerenv -o -f /.dockerenv 2024-06-18T02:17:44.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-18T02:17:44.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-18T02:17:44.788 DEBUG:teuthology.orchestra.run.smithi028:> uname -r 2024-06-18T02:17:44.788 DEBUG:teuthology.orchestra.run.smithi001:> uname -r 2024-06-18T02:17:44.845 INFO:teuthology.orchestra.run.smithi001.stdout:5.14.0-437.el9.x86_64 2024-06-18T02:17:44.846 INFO:teuthology.task.kernel:Running kernel on smithi001: 5.14.0-437.el9.x86_64 2024-06-18T02:17:44.846 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum install -y kernel 2024-06-18T02:17:44.847 INFO:teuthology.orchestra.run.smithi028.stdout:5.14.0-437.el9.x86_64 2024-06-18T02:17:44.847 INFO:teuthology.task.kernel:Running kernel on smithi028: 5.14.0-437.el9.x86_64 2024-06-18T02:17:44.847 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum install -y kernel 2024-06-18T02:17:46.880 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 8.1 MB 00:00 2024-06-18T02:17:48.374 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - BaseOS 4.3 MB/s | 8.1 MB 00:01 2024-06-18T02:17:48.872 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - AppStream 25 MB/s | 20 MB 00:00 2024-06-18T02:17:50.250 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - AppStream 28 MB/s | 20 MB 00:00 2024-06-18T02:17:55.764 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - CRB 3.8 MB/s | 6.3 MB 00:01 2024-06-18T02:17:56.648 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - CRB 1.6 MB/s | 6.3 MB 00:03 2024-06-18T02:17:57.477 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - Extras packages 37 kB/s | 17 kB 00:00 2024-06-18T02:17:58.382 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 17 kB 00:00 2024-06-18T02:17:59.712 INFO:teuthology.orchestra.run.smithi028.stdout:Extra Packages for Enterprise Linux 18 MB/s | 21 MB 00:01 2024-06-18T02:17:59.882 INFO:teuthology.orchestra.run.smithi001.stdout:Extra Packages for Enterprise Linux 9.5 MB/s | 21 MB 00:02 2024-06-18T02:18:05.168 INFO:teuthology.orchestra.run.smithi028.stdout:lab-extras 20 kB/s | 1.7 kB 00:00 2024-06-18T02:18:05.287 INFO:teuthology.orchestra.run.smithi001.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-06-18T02:18:07.315 INFO:teuthology.orchestra.run.smithi028.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-06-18T02:18:07.315 INFO:teuthology.orchestra.run.smithi028.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-06-18T02:18:07.408 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-06-18T02:18:07.413 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-06-18T02:18:07.413 INFO:teuthology.orchestra.run.smithi028.stdout: Package Architecture Version Repository Size 2024-06-18T02:18:07.413 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-06-18T02:18:07.413 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-06-18T02:18:07.413 INFO:teuthology.orchestra.run.smithi028.stdout: kernel x86_64 5.14.0-460.el9 baseos 1.1 M 2024-06-18T02:18:07.413 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-06-18T02:18:07.413 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-core x86_64 5.14.0-460.el9 baseos 16 M 2024-06-18T02:18:07.413 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-modules x86_64 5.14.0-460.el9 baseos 35 M 2024-06-18T02:18:07.413 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-modules-core x86_64 5.14.0-460.el9 baseos 29 M 2024-06-18T02:18:07.414 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:18:07.414 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-06-18T02:18:07.414 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-06-18T02:18:07.414 INFO:teuthology.orchestra.run.smithi028.stdout:Install 4 Packages 2024-06-18T02:18:07.414 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:18:07.414 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 81 M 2024-06-18T02:18:07.415 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 124 M 2024-06-18T02:18:07.415 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-06-18T02:18:07.452 INFO:teuthology.orchestra.run.smithi001.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-06-18T02:18:07.452 INFO:teuthology.orchestra.run.smithi001.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-06-18T02:18:07.545 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-06-18T02:18:07.552 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-06-18T02:18:07.552 INFO:teuthology.orchestra.run.smithi001.stdout: Package Architecture Version Repository Size 2024-06-18T02:18:07.552 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-06-18T02:18:07.552 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-06-18T02:18:07.552 INFO:teuthology.orchestra.run.smithi001.stdout: kernel x86_64 5.14.0-460.el9 baseos 1.1 M 2024-06-18T02:18:07.552 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-06-18T02:18:07.553 INFO:teuthology.orchestra.run.smithi001.stdout: kernel-core x86_64 5.14.0-460.el9 baseos 16 M 2024-06-18T02:18:07.553 INFO:teuthology.orchestra.run.smithi001.stdout: kernel-modules x86_64 5.14.0-460.el9 baseos 35 M 2024-06-18T02:18:07.553 INFO:teuthology.orchestra.run.smithi001.stdout: kernel-modules-core x86_64 5.14.0-460.el9 baseos 29 M 2024-06-18T02:18:07.553 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:18:07.553 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-06-18T02:18:07.553 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-06-18T02:18:07.553 INFO:teuthology.orchestra.run.smithi001.stdout:Install 4 Packages 2024-06-18T02:18:07.553 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:18:07.554 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 81 M 2024-06-18T02:18:07.554 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 124 M 2024-06-18T02:18:07.554 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-06-18T02:18:07.678 INFO:teuthology.orchestra.run.smithi001.stdout:[MIRROR] kernel-5.14.0-460.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-460.el9.x86_64.rpm (IP: 216.31.2.234) 2024-06-18T02:18:07.681 INFO:teuthology.orchestra.run.smithi001.stdout:[MIRROR] kernel-core-5.14.0-460.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-460.el9.x86_64.rpm (IP: 216.31.2.234) 2024-06-18T02:18:07.682 INFO:teuthology.orchestra.run.smithi001.stdout:[MIRROR] kernel-modules-5.14.0-460.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-460.el9.x86_64.rpm (IP: 216.31.2.234) 2024-06-18T02:18:07.815 INFO:teuthology.orchestra.run.smithi001.stdout:[MIRROR] kernel-5.14.0-460.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-460.el9.x86_64.rpm (IP: 216.31.2.234) 2024-06-18T02:18:07.821 INFO:teuthology.orchestra.run.smithi001.stdout:[MIRROR] kernel-core-5.14.0-460.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-460.el9.x86_64.rpm (IP: 216.31.2.234) 2024-06-18T02:18:07.846 INFO:teuthology.orchestra.run.smithi001.stdout:[MIRROR] kernel-modules-5.14.0-460.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-460.el9.x86_64.rpm (IP: 216.31.2.234) 2024-06-18T02:18:08.047 INFO:teuthology.orchestra.run.smithi001.stdout:(1/4): kernel-5.14.0-460.el9.x86_64.rpm 2.5 MB/s | 1.1 MB 00:00 2024-06-18T02:18:08.075 INFO:teuthology.orchestra.run.smithi001.stdout:[MIRROR] kernel-modules-core-5.14.0-460.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-460.el9.x86_64.rpm (IP: 216.31.2.234) 2024-06-18T02:18:08.104 INFO:teuthology.orchestra.run.smithi001.stdout:[MIRROR] kernel-modules-core-5.14.0-460.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-460.el9.x86_64.rpm (IP: 216.31.2.234) 2024-06-18T02:18:08.227 INFO:teuthology.orchestra.run.smithi028.stdout:(1/4): kernel-5.14.0-460.el9.x86_64.rpm 1.4 MB/s | 1.1 MB 00:00 2024-06-18T02:18:10.747 INFO:teuthology.orchestra.run.smithi001.stdout:(2/4): kernel-modules-core-5.14.0-460.el9.x86_6 11 MB/s | 29 MB 00:02 2024-06-18T02:18:12.364 INFO:teuthology.orchestra.run.smithi001.stdout:(3/4): kernel-core-5.14.0-460.el9.x86_64.rpm 3.4 MB/s | 16 MB 00:04 2024-06-18T02:18:17.688 INFO:teuthology.orchestra.run.smithi001.stdout:(4/4): kernel-modules-5.14.0-460.el9.x86_64.rpm 3.5 MB/s | 35 MB 00:10 2024-06-18T02:18:17.691 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-06-18T02:18:17.692 INFO:teuthology.orchestra.run.smithi001.stdout:Total 8.0 MB/s | 81 MB 00:10 2024-06-18T02:18:18.426 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-06-18T02:18:18.832 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-06-18T02:18:18.832 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-06-18T02:18:19.849 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-06-18T02:18:19.850 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-06-18T02:18:22.397 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-06-18T02:18:23.042 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : kernel-modules-core-5.14.0-460.el9.x86_64 1/4 2024-06-18T02:18:23.139 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : kernel-core-5.14.0-460.el9.x86_64 2/4 2024-06-18T02:18:23.959 INFO:teuthology.orchestra.run.smithi028.stdout:(2/4): kernel-core-5.14.0-460.el9.x86_64.rpm 1.0 MB/s | 16 MB 00:16 2024-06-18T02:18:25.961 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: kernel-core-5.14.0-460.el9.x86_64 2/4 2024-06-18T02:18:26.108 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : kernel-modules-5.14.0-460.el9.x86_64 3/4 2024-06-18T02:18:31.212 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: kernel-modules-5.14.0-460.el9.x86_64 3/4 2024-06-18T02:18:31.351 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : kernel-5.14.0-460.el9.x86_64 4/4 2024-06-18T02:18:35.916 INFO:teuthology.orchestra.run.smithi028.stdout:(3/4): kernel-modules-core-5.14.0-460.el9.x86_6 1.1 MB/s | 29 MB 00:27 2024-06-18T02:18:36.561 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: kernel-modules-core-5.14.0-460.el9.x86_64 4/4 2024-06-18T02:18:46.707 INFO:teuthology.orchestra.run.smithi028.stdout:(4/4): kernel-modules-5.14.0-460.el9.x86_64.rpm 908 kB/s | 35 MB 00:39 2024-06-18T02:18:46.708 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-06-18T02:18:46.708 INFO:teuthology.orchestra.run.smithi028.stdout:Total 2.1 MB/s | 81 MB 00:39 2024-06-18T02:18:47.445 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-06-18T02:18:47.869 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-06-18T02:18:47.869 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-06-18T02:18:48.855 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-06-18T02:18:48.855 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-06-18T02:18:51.391 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-06-18T02:18:52.037 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : kernel-modules-core-5.14.0-460.el9.x86_64 1/4 2024-06-18T02:18:52.116 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : kernel-core-5.14.0-460.el9.x86_64 2/4 2024-06-18T02:18:54.927 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-core-5.14.0-460.el9.x86_64 2/4 2024-06-18T02:18:55.070 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : kernel-modules-5.14.0-460.el9.x86_64 3/4 2024-06-18T02:19:00.183 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-modules-5.14.0-460.el9.x86_64 3/4 2024-06-18T02:19:00.235 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : kernel-5.14.0-460.el9.x86_64 4/4 2024-06-18T02:19:05.349 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-modules-core-5.14.0-460.el9.x86_64 4/4 2024-06-18T02:19:07.576 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: kernel-core-5.14.0-460.el9.x86_64 4/4 2024-06-18T02:19:07.618 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: kernel-modules-5.14.0-460.el9.x86_64 4/4 2024-06-18T02:19:07.978 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: kernel-5.14.0-460.el9.x86_64 4/4 2024-06-18T02:19:07.978 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : kernel-5.14.0-460.el9.x86_64 1/4 2024-06-18T02:19:07.978 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : kernel-core-5.14.0-460.el9.x86_64 2/4 2024-06-18T02:19:07.978 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : kernel-modules-5.14.0-460.el9.x86_64 3/4 2024-06-18T02:19:08.829 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : kernel-modules-core-5.14.0-460.el9.x86_64 4/4 2024-06-18T02:19:08.829 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:19:08.829 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-06-18T02:19:08.829 INFO:teuthology.orchestra.run.smithi001.stdout: kernel-5.14.0-460.el9.x86_64 2024-06-18T02:19:08.829 INFO:teuthology.orchestra.run.smithi001.stdout: kernel-core-5.14.0-460.el9.x86_64 2024-06-18T02:19:08.829 INFO:teuthology.orchestra.run.smithi001.stdout: kernel-modules-5.14.0-460.el9.x86_64 2024-06-18T02:19:08.829 INFO:teuthology.orchestra.run.smithi001.stdout: kernel-modules-core-5.14.0-460.el9.x86_64 2024-06-18T02:19:08.829 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:19:08.829 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-06-18T02:19:09.218 DEBUG:teuthology.orchestra.run.smithi001:> rpm -q kernel | sort -rV | head -n 1 2024-06-18T02:19:09.299 INFO:teuthology.orchestra.run.smithi001.stdout:kernel-5.14.0-460.el9.x86_64 2024-06-18T02:19:09.301 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-460.el9.x86_64 2024-06-18T02:19:09.301 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-460.el9.x86_64 2024-06-18T02:19:09.301 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-06-18T02:19:09.301 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-06-18T02:19:09.302 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-06-18T02:19:09.302 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi001.front.sepia.ceph.com, path=None, version=distro) 2024-06-18T02:19:09.302 DEBUG:teuthology.orchestra.run.smithi001:> rpm -q kernel | sort -rV | head -n 1 2024-06-18T02:19:09.356 INFO:teuthology.orchestra.run.smithi001.stdout:kernel-5.14.0-460.el9.x86_64 2024-06-18T02:19:09.357 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-460.el9.x86_64 2024-06-18T02:19:09.357 DEBUG:teuthology.orchestra.run.smithi001:> sudo rpm -qi grub2-tools 2024-06-18T02:19:09.393 INFO:teuthology.orchestra.run.smithi001.stdout:Name : grub2-tools 2024-06-18T02:19:09.393 INFO:teuthology.orchestra.run.smithi001.stdout:Epoch : 1 2024-06-18T02:19:09.393 INFO:teuthology.orchestra.run.smithi001.stdout:Version : 2.06 2024-06-18T02:19:09.393 INFO:teuthology.orchestra.run.smithi001.stdout:Release : 68.el9 2024-06-18T02:19:09.393 INFO:teuthology.orchestra.run.smithi001.stdout:Architecture: x86_64 2024-06-18T02:19:09.393 INFO:teuthology.orchestra.run.smithi001.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-06-18T02:19:09.393 INFO:teuthology.orchestra.run.smithi001.stdout:Group : Unspecified 2024-06-18T02:19:09.393 INFO:teuthology.orchestra.run.smithi001.stdout:Size : 8264254 2024-06-18T02:19:09.394 INFO:teuthology.orchestra.run.smithi001.stdout:License : GPLv3+ 2024-06-18T02:19:09.394 INFO:teuthology.orchestra.run.smithi001.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-06-18T02:19:09.394 INFO:teuthology.orchestra.run.smithi001.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-06-18T02:19:09.394 INFO:teuthology.orchestra.run.smithi001.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-06-18T02:19:09.394 INFO:teuthology.orchestra.run.smithi001.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-06-18T02:19:09.394 INFO:teuthology.orchestra.run.smithi001.stdout:Packager : builder@centos.org 2024-06-18T02:19:09.394 INFO:teuthology.orchestra.run.smithi001.stdout:Vendor : CentOS 2024-06-18T02:19:09.394 INFO:teuthology.orchestra.run.smithi001.stdout:URL : http://www.gnu.org/software/grub/ 2024-06-18T02:19:09.394 INFO:teuthology.orchestra.run.smithi001.stdout:Summary : Support tools for GRUB. 2024-06-18T02:19:09.394 INFO:teuthology.orchestra.run.smithi001.stdout:Description : 2024-06-18T02:19:09.394 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:19:09.394 INFO:teuthology.orchestra.run.smithi001.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-06-18T02:19:09.394 INFO:teuthology.orchestra.run.smithi001.stdout:customizable bootloader with modular architecture. It supports a rich 2024-06-18T02:19:09.395 INFO:teuthology.orchestra.run.smithi001.stdout:variety of kernel formats, file systems, computer architectures and 2024-06-18T02:19:09.395 INFO:teuthology.orchestra.run.smithi001.stdout:hardware devices. 2024-06-18T02:19:09.395 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:19:09.395 INFO:teuthology.orchestra.run.smithi001.stdout:This subpackage provides tools for support of all platforms. 2024-06-18T02:19:09.396 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-06-18T02:19:09.396 INFO:teuthology.task.kernel:Updating grub on smithi001 to boot 5.14.0-460.el9.x86_64 2024-06-18T02:19:09.396 DEBUG:teuthology.orchestra.run.smithi001:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-06-18T02:19:10.009 INFO:teuthology.orchestra.run.smithi001.stderr:Generating grub configuration file ... 2024-06-18T02:19:11.436 INFO:teuthology.orchestra.run.smithi001.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-06-18T02:19:11.460 INFO:teuthology.orchestra.run.smithi001.stderr:done 2024-06-18T02:19:11.463 DEBUG:teuthology.orchestra.run.smithi001:> mktemp 2024-06-18T02:19:11.484 INFO:teuthology.orchestra.run.smithi001.stdout:/tmp/tmp.4mzsasAY4G 2024-06-18T02:19:11.485 DEBUG:teuthology.orchestra.run.smithi001:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.4mzsasAY4G 2024-06-18T02:19:11.551 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 0666 /tmp/tmp.4mzsasAY4G 2024-06-18T02:19:11.684 DEBUG:teuthology.orchestra.remote:smithi001:/tmp/tmp.4mzsasAY4G is 6KB 2024-06-18T02:19:11.733 DEBUG:teuthology.orchestra.run.smithi001:> rm -fr /tmp/tmp.4mzsasAY4G 2024-06-18T02:19:11.748 DEBUG:teuthology.orchestra.run.smithi001:> sudo /bin/ls /boot/loader/entries || true 2024-06-18T02:19:11.810 INFO:teuthology.orchestra.run.smithi001.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-06-18T02:19:11.811 INFO:teuthology.orchestra.run.smithi001.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-06-18T02:19:11.811 INFO:teuthology.orchestra.run.smithi001.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-06-18T02:19:11.811 INFO:teuthology.orchestra.run.smithi001.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-460.el9.x86_64.conf 2024-06-18T02:19:11.812 DEBUG:teuthology.orchestra.run.smithi001:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-460.el9.x86_64 2024-06-18T02:19:11.966 DEBUG:teuthology.orchestra.run.smithi001:> sudo shutdown -r now 2024-06-18T02:19:36.385 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-core-5.14.0-460.el9.x86_64 4/4 2024-06-18T02:19:36.427 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-modules-5.14.0-460.el9.x86_64 4/4 2024-06-18T02:19:36.859 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-5.14.0-460.el9.x86_64 4/4 2024-06-18T02:19:36.859 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : kernel-5.14.0-460.el9.x86_64 1/4 2024-06-18T02:19:36.859 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : kernel-core-5.14.0-460.el9.x86_64 2/4 2024-06-18T02:19:36.859 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : kernel-modules-5.14.0-460.el9.x86_64 3/4 2024-06-18T02:19:37.678 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : kernel-modules-core-5.14.0-460.el9.x86_64 4/4 2024-06-18T02:19:37.678 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:19:37.678 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-06-18T02:19:37.678 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-5.14.0-460.el9.x86_64 2024-06-18T02:19:37.678 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-core-5.14.0-460.el9.x86_64 2024-06-18T02:19:37.679 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-modules-5.14.0-460.el9.x86_64 2024-06-18T02:19:37.679 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-modules-core-5.14.0-460.el9.x86_64 2024-06-18T02:19:37.679 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:19:37.679 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-06-18T02:19:38.041 DEBUG:teuthology.orchestra.run.smithi028:> rpm -q kernel | sort -rV | head -n 1 2024-06-18T02:19:38.118 INFO:teuthology.orchestra.run.smithi028.stdout:kernel-5.14.0-460.el9.x86_64 2024-06-18T02:19:38.119 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-460.el9.x86_64 2024-06-18T02:19:38.119 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-460.el9.x86_64 2024-06-18T02:19:38.119 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-06-18T02:19:38.119 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-06-18T02:19:38.119 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-06-18T02:19:38.119 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi028.front.sepia.ceph.com, path=None, version=distro) 2024-06-18T02:19:38.119 DEBUG:teuthology.orchestra.run.smithi028:> rpm -q kernel | sort -rV | head -n 1 2024-06-18T02:19:38.173 INFO:teuthology.orchestra.run.smithi028.stdout:kernel-5.14.0-460.el9.x86_64 2024-06-18T02:19:38.174 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-460.el9.x86_64 2024-06-18T02:19:38.174 DEBUG:teuthology.orchestra.run.smithi028:> sudo rpm -qi grub2-tools 2024-06-18T02:19:38.206 INFO:teuthology.orchestra.run.smithi028.stdout:Name : grub2-tools 2024-06-18T02:19:38.206 INFO:teuthology.orchestra.run.smithi028.stdout:Epoch : 1 2024-06-18T02:19:38.206 INFO:teuthology.orchestra.run.smithi028.stdout:Version : 2.06 2024-06-18T02:19:38.206 INFO:teuthology.orchestra.run.smithi028.stdout:Release : 68.el9 2024-06-18T02:19:38.206 INFO:teuthology.orchestra.run.smithi028.stdout:Architecture: x86_64 2024-06-18T02:19:38.206 INFO:teuthology.orchestra.run.smithi028.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-06-18T02:19:38.206 INFO:teuthology.orchestra.run.smithi028.stdout:Group : Unspecified 2024-06-18T02:19:38.206 INFO:teuthology.orchestra.run.smithi028.stdout:Size : 8264254 2024-06-18T02:19:38.206 INFO:teuthology.orchestra.run.smithi028.stdout:License : GPLv3+ 2024-06-18T02:19:38.206 INFO:teuthology.orchestra.run.smithi028.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-06-18T02:19:38.206 INFO:teuthology.orchestra.run.smithi028.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-06-18T02:19:38.207 INFO:teuthology.orchestra.run.smithi028.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-06-18T02:19:38.207 INFO:teuthology.orchestra.run.smithi028.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-06-18T02:19:38.207 INFO:teuthology.orchestra.run.smithi028.stdout:Packager : builder@centos.org 2024-06-18T02:19:38.207 INFO:teuthology.orchestra.run.smithi028.stdout:Vendor : CentOS 2024-06-18T02:19:38.207 INFO:teuthology.orchestra.run.smithi028.stdout:URL : http://www.gnu.org/software/grub/ 2024-06-18T02:19:38.207 INFO:teuthology.orchestra.run.smithi028.stdout:Summary : Support tools for GRUB. 2024-06-18T02:19:38.207 INFO:teuthology.orchestra.run.smithi028.stdout:Description : 2024-06-18T02:19:38.207 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:19:38.207 INFO:teuthology.orchestra.run.smithi028.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-06-18T02:19:38.207 INFO:teuthology.orchestra.run.smithi028.stdout:customizable bootloader with modular architecture. It supports a rich 2024-06-18T02:19:38.207 INFO:teuthology.orchestra.run.smithi028.stdout:variety of kernel formats, file systems, computer architectures and 2024-06-18T02:19:38.207 INFO:teuthology.orchestra.run.smithi028.stdout:hardware devices. 2024-06-18T02:19:38.207 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:19:38.207 INFO:teuthology.orchestra.run.smithi028.stdout:This subpackage provides tools for support of all platforms. 2024-06-18T02:19:38.208 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-06-18T02:19:38.208 INFO:teuthology.task.kernel:Updating grub on smithi028 to boot 5.14.0-460.el9.x86_64 2024-06-18T02:19:38.208 DEBUG:teuthology.orchestra.run.smithi028:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-06-18T02:19:38.852 INFO:teuthology.orchestra.run.smithi028.stderr:Generating grub configuration file ... 2024-06-18T02:19:40.274 INFO:teuthology.orchestra.run.smithi028.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-06-18T02:19:40.312 INFO:teuthology.orchestra.run.smithi028.stderr:done 2024-06-18T02:19:40.314 DEBUG:teuthology.orchestra.run.smithi028:> mktemp 2024-06-18T02:19:40.370 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/tmp.CTGU6c9Acg 2024-06-18T02:19:40.371 DEBUG:teuthology.orchestra.run.smithi028:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.CTGU6c9Acg 2024-06-18T02:19:40.503 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 0666 /tmp/tmp.CTGU6c9Acg 2024-06-18T02:19:40.601 DEBUG:teuthology.orchestra.remote:smithi028:/tmp/tmp.CTGU6c9Acg is 6KB 2024-06-18T02:19:40.649 DEBUG:teuthology.orchestra.run.smithi028:> rm -fr /tmp/tmp.CTGU6c9Acg 2024-06-18T02:19:40.664 DEBUG:teuthology.orchestra.run.smithi028:> sudo /bin/ls /boot/loader/entries || true 2024-06-18T02:19:40.726 INFO:teuthology.orchestra.run.smithi028.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-06-18T02:19:40.726 INFO:teuthology.orchestra.run.smithi028.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-06-18T02:19:40.726 INFO:teuthology.orchestra.run.smithi028.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-06-18T02:19:40.726 INFO:teuthology.orchestra.run.smithi028.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-460.el9.x86_64.conf 2024-06-18T02:19:40.728 DEBUG:teuthology.orchestra.run.smithi028:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-460.el9.x86_64 2024-06-18T02:19:40.861 DEBUG:teuthology.orchestra.run.smithi028:> sudo shutdown -r now 2024-06-18T02:19:41.970 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-06-18T02:19:41.970 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-06-18T02:19:41.971 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-18T02:20:00.414 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.1 2024-06-18T02:20:09.422 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-06-18T02:20:09.423 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-18T02:20:10.865 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-06-18T02:20:10.865 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-06-18T02:20:10.866 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-18T02:20:29.347 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.28 2024-06-18T02:20:38.350 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-06-18T02:20:38.351 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-18T02:20:56.741 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.28 2024-06-18T02:21:08.753 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-06-18T02:21:08.754 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-18T02:21:09.426 DEBUG:teuthology.orchestra.remote:timed out 2024-06-18T02:21:11.806 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.28 2024-06-18T02:21:21.435 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-06-18T02:21:21.436 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-18T02:21:21.716 DEBUG:teuthology.orchestra.run.smithi001:> true 2024-06-18T02:21:22.070 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-06-18T02:21:22.070 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.14.0-460.el9.x86_64"... 2024-06-18T02:21:22.070 DEBUG:teuthology.orchestra.run.smithi001:> uname -r 2024-06-18T02:21:22.086 INFO:teuthology.orchestra.run.smithi001.stdout:5.14.0-460.el9.x86_64 2024-06-18T02:21:22.086 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-460.el9.x86_64 vs 5.14.0-460.el9.x86_64 2024-06-18T02:21:22.086 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-06-18T02:21:22.087 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-06-18T02:21:23.087 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-06-18T02:21:23.087 DEBUG:teuthology.orchestra.run.smithi001:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-06-18T02:21:23.163 INFO:teuthology.orchestra.run.smithi001.stdout:ttyS1 2024-06-18T02:21:23.203 DEBUG:teuthology.parallel:result is None 2024-06-18T02:21:26.807 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-06-18T02:21:26.808 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-18T02:21:29.886 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.28 2024-06-18T02:21:47.887 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-06-18T02:21:47.888 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-18T02:21:48.196 DEBUG:teuthology.orchestra.run.smithi028:> true 2024-06-18T02:21:48.585 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-06-18T02:21:48.585 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.14.0-460.el9.x86_64"... 2024-06-18T02:21:48.586 DEBUG:teuthology.orchestra.run.smithi028:> uname -r 2024-06-18T02:21:48.603 INFO:teuthology.orchestra.run.smithi028.stdout:5.14.0-460.el9.x86_64 2024-06-18T02:21:48.603 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-460.el9.x86_64 vs 5.14.0-460.el9.x86_64 2024-06-18T02:21:48.603 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-06-18T02:21:48.603 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-06-18T02:21:49.604 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-06-18T02:21:49.604 DEBUG:teuthology.orchestra.run.smithi028:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-06-18T02:21:49.686 INFO:teuthology.orchestra.run.smithi028.stdout:ttyS1 2024-06-18T02:21:49.719 DEBUG:teuthology.parallel:result is None 2024-06-18T02:21:49.719 INFO:teuthology.run_tasks:Running task internal.base... 2024-06-18T02:21:49.726 INFO:teuthology.task.internal:Creating test directory... 2024-06-18T02:21:49.726 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-06-18T02:21:49.729 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-06-18T02:21:49.748 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-06-18T02:21:49.754 INFO:teuthology.run_tasks:Running task internal.archive... 2024-06-18T02:21:49.763 INFO:teuthology.task.internal:Creating archive directory... 2024-06-18T02:21:49.764 DEBUG:teuthology.orchestra.run.smithi001:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-06-18T02:21:49.790 DEBUG:teuthology.orchestra.run.smithi028:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-06-18T02:21:49.826 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-06-18T02:21:49.833 INFO:teuthology.task.internal:Enabling coredump saving... 2024-06-18T02:21:49.833 DEBUG:teuthology.orchestra.run.smithi001:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-06-18T02:21:49.864 DEBUG:teuthology.orchestra.run.smithi028:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-06-18T02:21:49.895 INFO:teuthology.orchestra.run.smithi001.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-18T02:21:49.897 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-18T02:21:49.907 INFO:teuthology.orchestra.run.smithi001.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-18T02:21:49.909 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-18T02:21:49.911 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-06-18T02:21:49.919 INFO:teuthology.task.internal:Configuring sudo... 2024-06-18T02:21:49.919 DEBUG:teuthology.orchestra.run.smithi001:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-06-18T02:21:49.951 DEBUG:teuthology.orchestra.run.smithi028:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-06-18T02:21:49.981 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-06-18T02:21:49.989 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-06-18T02:21:49.989 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-06-18T02:21:50.021 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-06-18T02:21:50.038 DEBUG:teuthology.orchestra.run.smithi001:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-18T02:21:50.122 DEBUG:teuthology.orchestra.run.smithi001:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-18T02:21:50.170 DEBUG:teuthology.orchestra.run.smithi001:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-18T02:21:50.254 DEBUG:teuthology.orchestra.run.smithi001:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-18T02:21:50.295 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-06-18T02:21:50.296 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-06-18T02:21:50.362 DEBUG:teuthology.orchestra.run.smithi028:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-18T02:21:50.409 DEBUG:teuthology.orchestra.run.smithi028:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-18T02:21:50.463 DEBUG:teuthology.orchestra.run.smithi028:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-18T02:21:50.548 DEBUG:teuthology.orchestra.run.smithi028:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-18T02:21:50.593 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-06-18T02:21:50.593 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-06-18T02:21:50.661 DEBUG:teuthology.orchestra.run.smithi001:> sudo service rsyslog restart 2024-06-18T02:21:50.663 DEBUG:teuthology.orchestra.run.smithi028:> sudo service rsyslog restart 2024-06-18T02:21:50.696 INFO:teuthology.orchestra.run.smithi001.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-18T02:21:50.749 INFO:teuthology.orchestra.run.smithi028.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-18T02:21:51.166 INFO:teuthology.run_tasks:Running task internal.timer... 2024-06-18T02:21:51.174 INFO:teuthology.task.internal:Starting timer... 2024-06-18T02:21:51.174 INFO:teuthology.run_tasks:Running task pcp... 2024-06-18T02:21:51.184 INFO:teuthology.run_tasks:Running task selinux... 2024-06-18T02:21:51.192 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-06-18T02:21:51.192 DEBUG:teuthology.orchestra.run.smithi001:> sudo service auditd rotate 2024-06-18T02:21:51.272 INFO:teuthology.orchestra.run.smithi001.stdout:Rotating logs: 2024-06-18T02:21:51.274 DEBUG:teuthology.orchestra.run.smithi028:> sudo service auditd rotate 2024-06-18T02:21:51.324 INFO:teuthology.orchestra.run.smithi028.stdout:Rotating logs: 2024-06-18T02:21:51.325 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-06-18T02:21:51.326 DEBUG:teuthology.orchestra.run.smithi001:> /usr/sbin/getenforce 2024-06-18T02:21:51.365 INFO:teuthology.orchestra.run.smithi001.stdout:Permissive 2024-06-18T02:21:51.366 DEBUG:teuthology.orchestra.run.smithi028:> /usr/sbin/getenforce 2024-06-18T02:21:51.400 INFO:teuthology.orchestra.run.smithi028.stdout:Permissive 2024-06-18T02:21:51.400 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi001.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi028.front.sepia.ceph.com': 'permissive'} 2024-06-18T02:21:51.400 DEBUG:teuthology.orchestra.run.smithi001:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-18T02:21:51.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-18T02:21:51.435 DEBUG:teuthology.orchestra.run.smithi028:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-18T02:21:51.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-18T02:21:51.469 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-06-18T02:21:51.469 DEBUG:teuthology.orchestra.run.smithi001:> sudo /usr/sbin/setenforce permissive 2024-06-18T02:21:51.510 DEBUG:teuthology.orchestra.run.smithi028:> sudo /usr/sbin/setenforce permissive 2024-06-18T02:21:51.540 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-06-18T02:21:51.550 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-06-18T02:21:51.560 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-06-18T02:21:51.561 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi001.front.sepia.ceph.com,smithi028.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-06-18T02:26:23.694 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi001.front.sepia.ceph.com'), Remote(name='ubuntu@smithi028.front.sepia.ceph.com')] 2024-06-18T02:26:23.695 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-06-18T02:26:23.696 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-18T02:26:23.775 DEBUG:teuthology.orchestra.run.smithi001:> true 2024-06-18T02:26:23.850 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-06-18T02:26:23.851 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-06-18T02:26:23.851 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-18T02:26:23.925 DEBUG:teuthology.orchestra.run.smithi028:> true 2024-06-18T02:26:24.007 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-06-18T02:26:24.008 INFO:teuthology.run_tasks:Running task clock... 2024-06-18T02:26:24.018 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-06-18T02:26:24.018 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-06-18T02:26:24.019 DEBUG:teuthology.orchestra.run.smithi001:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-18T02:26:24.021 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-06-18T02:26:24.021 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-18T02:26:24.051 INFO:teuthology.orchestra.run.smithi001.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-06-18T02:26:24.067 INFO:teuthology.orchestra.run.smithi001.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-06-18T02:26:24.079 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-06-18T02:26:24.094 INFO:teuthology.orchestra.run.smithi001.stderr:sudo: ntpd: command not found 2024-06-18T02:26:24.094 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-06-18T02:26:24.105 INFO:teuthology.orchestra.run.smithi001.stdout:506 Cannot talk to daemon 2024-06-18T02:26:24.120 INFO:teuthology.orchestra.run.smithi001.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-06-18T02:26:24.120 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: ntpd: command not found 2024-06-18T02:26:24.132 INFO:teuthology.orchestra.run.smithi028.stdout:506 Cannot talk to daemon 2024-06-18T02:26:24.135 INFO:teuthology.orchestra.run.smithi001.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-06-18T02:26:24.146 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-06-18T02:26:24.161 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-06-18T02:26:24.190 INFO:teuthology.orchestra.run.smithi001.stderr:bash: line 1: ntpq: command not found 2024-06-18T02:26:24.193 INFO:teuthology.orchestra.run.smithi001.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-18T02:26:24.193 INFO:teuthology.orchestra.run.smithi001.stdout:=============================================================================== 2024-06-18T02:26:24.194 INFO:teuthology.orchestra.run.smithi001.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-18T02:26:24.194 INFO:teuthology.orchestra.run.smithi001.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-18T02:26:24.194 INFO:teuthology.orchestra.run.smithi001.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-18T02:26:24.194 INFO:teuthology.orchestra.run.smithi001.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-18T02:26:24.217 INFO:teuthology.orchestra.run.smithi028.stderr:bash: line 1: ntpq: command not found 2024-06-18T02:26:24.222 INFO:teuthology.orchestra.run.smithi028.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-18T02:26:24.222 INFO:teuthology.orchestra.run.smithi028.stdout:=============================================================================== 2024-06-18T02:26:24.222 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-18T02:26:24.222 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-18T02:26:24.222 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-18T02:26:24.222 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-18T02:26:24.223 INFO:teuthology.run_tasks:Running task install... 2024-06-18T02:26:24.230 DEBUG:teuthology.task.install:project ceph 2024-06-18T02:26:24.230 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '3781e3a592387d16e8020f5db99cdee4198f6d00'}} 2024-06-18T02:26:24.230 DEBUG:teuthology.task.install:config {'branch': 'quincy', 'exclude_packages': ['ceph-volume'], 'flavor': 'default', 'sha1': '3781e3a592387d16e8020f5db99cdee4198f6d00'} 2024-06-18T02:26:24.230 INFO:teuthology.task.install:Using flavor: default 2024-06-18T02:26:24.235 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-06-18T02:26:24.235 INFO:teuthology.task.install:extra packages: [] 2024-06-18T02:26:24.235 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3781e3a592387d16e8020f5db99cdee4198f6d00', 'tag': None, 'wait_for_package': False} 2024-06-18T02:26:24.236 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-18T02:26:24.236 INFO:teuthology.packaging:ref: None 2024-06-18T02:26:24.236 INFO:teuthology.packaging:tag: None 2024-06-18T02:26:24.236 INFO:teuthology.packaging:branch: quincy 2024-06-18T02:26:24.236 INFO:teuthology.packaging:sha1: 3781e3a592387d16e8020f5db99cdee4198f6d00 2024-06-18T02:26:24.236 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-06-18T02:26:24.238 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3781e3a592387d16e8020f5db99cdee4198f6d00', 'tag': None, 'wait_for_package': False} 2024-06-18T02:26:24.238 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-18T02:26:24.238 INFO:teuthology.packaging:ref: None 2024-06-18T02:26:24.238 INFO:teuthology.packaging:tag: None 2024-06-18T02:26:24.238 INFO:teuthology.packaging:branch: quincy 2024-06-18T02:26:24.238 INFO:teuthology.packaging:sha1: 3781e3a592387d16e8020f5db99cdee4198f6d00 2024-06-18T02:26:24.238 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-06-18T02:26:24.533 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/quincy/3fe410552b224c0fbe57dfbac3e18568c0f8fef6/centos/9/flavors/default/ 2024-06-18T02:26:24.534 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1012.g3fe41055 2024-06-18T02:26:24.557 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/quincy/3fe410552b224c0fbe57dfbac3e18568c0f8fef6/centos/9/flavors/default/ 2024-06-18T02:26:24.558 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1012.g3fe41055 2024-06-18T02:26:24.647 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/quincy/3fe410552b224c0fbe57dfbac3e18568c0f8fef6/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/quincy/3fe410552b224c0fbe57dfbac3e18568c0f8fef6/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/quincy/3fe410552b224c0fbe57dfbac3e18568c0f8fef6/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-06-18T02:26:24.647 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-06-18T02:26:24.647 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-06-18T02:26:24.675 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-06-18T02:26:24.675 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-18T02:26:24.675 INFO:teuthology.packaging:ref: None 2024-06-18T02:26:24.675 INFO:teuthology.packaging:tag: None 2024-06-18T02:26:24.675 INFO:teuthology.packaging:branch: quincy 2024-06-18T02:26:24.675 INFO:teuthology.packaging:sha1: 3781e3a592387d16e8020f5db99cdee4198f6d00 2024-06-18T02:26:24.675 DEBUG:teuthology.orchestra.run.smithi001:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-06-18T02:26:24.725 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/quincy/3fe410552b224c0fbe57dfbac3e18568c0f8fef6/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/quincy/3fe410552b224c0fbe57dfbac3e18568c0f8fef6/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/quincy/3fe410552b224c0fbe57dfbac3e18568c0f8fef6/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-06-18T02:26:24.725 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-06-18T02:26:24.725 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-06-18T02:26:24.742 DEBUG:teuthology.orchestra.run.smithi001:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-06-18T02:26:24.752 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-06-18T02:26:24.752 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-18T02:26:24.752 INFO:teuthology.packaging:ref: None 2024-06-18T02:26:24.753 INFO:teuthology.packaging:tag: None 2024-06-18T02:26:24.753 INFO:teuthology.packaging:branch: quincy 2024-06-18T02:26:24.753 INFO:teuthology.packaging:sha1: 3781e3a592387d16e8020f5db99cdee4198f6d00 2024-06-18T02:26:24.753 DEBUG:teuthology.orchestra.run.smithi028:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-06-18T02:26:24.819 DEBUG:teuthology.orchestra.run.smithi001:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-06-18T02:26:24.820 DEBUG:teuthology.orchestra.run.smithi028:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-06-18T02:26:24.885 INFO:teuthology.orchestra.run.smithi001.stdout:check_obsoletes = 1 2024-06-18T02:26:24.886 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum clean all 2024-06-18T02:26:24.896 DEBUG:teuthology.orchestra.run.smithi028:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-06-18T02:26:24.961 INFO:teuthology.orchestra.run.smithi028.stdout:check_obsoletes = 1 2024-06-18T02:26:24.963 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum clean all 2024-06-18T02:26:25.123 INFO:teuthology.orchestra.run.smithi001.stdout:45 files removed 2024-06-18T02:26:25.151 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-radosgw 2024-06-18T02:26:25.199 INFO:teuthology.orchestra.run.smithi028.stdout:45 files removed 2024-06-18T02:26:25.226 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-radosgw 2024-06-18T02:26:25.796 INFO:teuthology.orchestra.run.smithi001.stdout:ceph packages for x86_64 218 kB/s | 77 kB 00:00 2024-06-18T02:26:25.922 INFO:teuthology.orchestra.run.smithi028.stdout:ceph packages for x86_64 194 kB/s | 77 kB 00:00 2024-06-18T02:26:26.188 INFO:teuthology.orchestra.run.smithi001.stdout:ceph noarch packages 32 kB/s | 11 kB 00:00 2024-06-18T02:26:26.289 INFO:teuthology.orchestra.run.smithi028.stdout:ceph noarch packages 34 kB/s | 11 kB 00:00 2024-06-18T02:26:26.530 INFO:teuthology.orchestra.run.smithi001.stdout:ceph source packages 6.0 kB/s | 1.8 kB 00:00 2024-06-18T02:26:26.665 INFO:teuthology.orchestra.run.smithi028.stdout:ceph source packages 5.4 kB/s | 1.8 kB 00:00 2024-06-18T02:26:27.282 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.1 MB 00:00 2024-06-18T02:26:31.275 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - BaseOS 1.7 MB/s | 8.1 MB 00:04 2024-06-18T02:26:33.133 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - AppStream 4.2 MB/s | 20 MB 00:04 2024-06-18T02:26:37.434 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.3 MB 00:00 2024-06-18T02:26:38.968 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - Extras packages 64 kB/s | 17 kB 00:00 2024-06-18T02:26:39.939 INFO:teuthology.orchestra.run.smithi028.stdout:Extra Packages for Enterprise Linux 26 MB/s | 21 MB 00:00 2024-06-18T02:26:45.436 INFO:teuthology.orchestra.run.smithi028.stdout:lab-extras 49 kB/s | 1.7 kB 00:00 2024-06-18T02:26:47.946 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-06-18T02:26:47.949 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-06-18T02:26:47.949 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2024-06-18T02:26:47.949 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-06-18T02:26:47.949 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-06-18T02:26:47.949 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-radosgw x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 14 M 2024-06-18T02:26:47.949 INFO:teuthology.orchestra.run.smithi028.stdout:Upgrading: 2024-06-18T02:26:47.949 INFO:teuthology.orchestra.run.smithi028.stdout: librados2 x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 3.1 M 2024-06-18T02:26:47.949 INFO:teuthology.orchestra.run.smithi028.stdout: librbd1 x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 2.9 M 2024-06-18T02:26:47.949 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-06-18T02:26:47.949 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-base x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 5.0 M 2024-06-18T02:26:47.949 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-common x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 22 M 2024-06-18T02:26:47.950 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-selinux x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 25 k 2024-06-18T02:26:47.950 INFO:teuthology.orchestra.run.smithi028.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-06-18T02:26:47.950 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs2 x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 637 k 2024-06-18T02:26:47.950 INFO:teuthology.orchestra.run.smithi028.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-06-18T02:26:47.950 INFO:teuthology.orchestra.run.smithi028.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-06-18T02:26:47.950 INFO:teuthology.orchestra.run.smithi028.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-06-18T02:26:47.950 INFO:teuthology.orchestra.run.smithi028.stdout: libradosstriper1 x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 458 k 2024-06-18T02:26:47.950 INFO:teuthology.orchestra.run.smithi028.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-06-18T02:26:47.950 INFO:teuthology.orchestra.run.smithi028.stdout: librgw2 x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 4.9 M 2024-06-18T02:26:47.950 INFO:teuthology.orchestra.run.smithi028.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-06-18T02:26:47.950 INFO:teuthology.orchestra.run.smithi028.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-06-18T02:26:47.951 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-argparse x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 45 k 2024-06-18T02:26:47.951 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-common x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 118 k 2024-06-18T02:26:47.951 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cephfs x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 156 k 2024-06-18T02:26:47.951 INFO:teuthology.orchestra.run.smithi028.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-06-18T02:26:47.951 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rados x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 320 k 2024-06-18T02:26:47.951 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rbd x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 293 k 2024-06-18T02:26:47.951 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rgw x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 99 k 2024-06-18T02:26:47.951 INFO:teuthology.orchestra.run.smithi028.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-06-18T02:26:47.951 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:26:47.951 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-06-18T02:26:47.951 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-06-18T02:26:47.951 INFO:teuthology.orchestra.run.smithi028.stdout:Install 22 Packages 2024-06-18T02:26:47.951 INFO:teuthology.orchestra.run.smithi028.stdout:Upgrade 2 Packages 2024-06-18T02:26:47.951 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:26:47.952 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 58 M 2024-06-18T02:26:47.952 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-06-18T02:26:48.605 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 20 MB 00:16 2024-06-18T02:26:49.262 INFO:teuthology.orchestra.run.smithi028.stdout:(1/24): ceph-radosgw-17.2.7-1012.g3fe41055.el9. 14 MB/s | 14 MB 00:00 2024-06-18T02:26:49.313 INFO:teuthology.orchestra.run.smithi028.stdout:(2/24): ceph-selinux-17.2.7-1012.g3fe41055.el9. 504 kB/s | 25 kB 00:00 2024-06-18T02:26:49.396 INFO:teuthology.orchestra.run.smithi028.stdout:(3/24): libcephfs2-17.2.7-1012.g3fe41055.el9.x8 7.5 MB/s | 637 kB 00:00 2024-06-18T02:26:49.455 INFO:teuthology.orchestra.run.smithi028.stdout:(4/24): libradosstriper1-17.2.7-1012.g3fe41055. 7.6 MB/s | 458 kB 00:00 2024-06-18T02:26:49.606 INFO:teuthology.orchestra.run.smithi028.stdout:(5/24): ceph-base-17.2.7-1012.g3fe41055.el9.x86 3.7 MB/s | 5.0 MB 00:01 2024-06-18T02:26:49.731 INFO:teuthology.orchestra.run.smithi028.stdout:(6/24): librgw2-17.2.7-1012.g3fe41055.el9.x86_6 18 MB/s | 4.9 MB 00:00 2024-06-18T02:26:49.756 INFO:teuthology.orchestra.run.smithi028.stdout:(7/24): python3-ceph-argparse-17.2.7-1012.g3fe4 300 kB/s | 45 kB 00:00 2024-06-18T02:26:49.781 INFO:teuthology.orchestra.run.smithi028.stdout:(8/24): python3-ceph-common-17.2.7-1012.g3fe410 2.3 MB/s | 118 kB 00:00 2024-06-18T02:26:49.823 INFO:teuthology.orchestra.run.smithi028.stdout:(9/24): python3-cephfs-17.2.7-1012.g3fe41055.el 2.3 MB/s | 156 kB 00:00 2024-06-18T02:26:49.848 INFO:teuthology.orchestra.run.smithi028.stdout:(10/24): python3-rados-17.2.7-1012.g3fe41055.el 4.7 MB/s | 320 kB 00:00 2024-06-18T02:26:49.891 INFO:teuthology.orchestra.run.smithi028.stdout:(11/24): python3-rbd-17.2.7-1012.g3fe41055.el9. 4.3 MB/s | 293 kB 00:00 2024-06-18T02:26:49.916 INFO:teuthology.orchestra.run.smithi028.stdout:(12/24): python3-rgw-17.2.7-1012.g3fe41055.el9. 1.4 MB/s | 99 kB 00:00 2024-06-18T02:26:50.008 INFO:teuthology.orchestra.run.smithi028.stdout:(13/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 494 kB/s | 45 kB 00:00 2024-06-18T02:26:50.041 INFO:teuthology.orchestra.run.smithi028.stdout:(14/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 160 kB 00:00 2024-06-18T02:26:50.100 INFO:teuthology.orchestra.run.smithi028.stdout:(15/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.9 MB/s | 292 kB 00:00 2024-06-18T02:26:50.125 INFO:teuthology.orchestra.run.smithi028.stdout:(16/24): librdkafka-1.6.1-102.el9.x86_64.rpm 5.5 MB/s | 662 kB 00:00 2024-06-18T02:26:50.151 INFO:teuthology.orchestra.run.smithi028.stdout:(17/24): python3-prettytable-0.7.2-27.el9.noarc 838 kB/s | 42 kB 00:00 2024-06-18T02:26:50.251 INFO:teuthology.orchestra.run.smithi028.stdout:(18/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 2.4 MB/s | 309 kB 00:00 2024-06-18T02:26:50.276 INFO:teuthology.orchestra.run.smithi028.stdout:(19/24): liboath-2.6.7-2.el9.x86_64.rpm 388 kB/s | 49 kB 00:00 2024-06-18T02:26:50.301 INFO:teuthology.orchestra.run.smithi028.stdout:(20/24): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-06-18T02:26:50.352 INFO:teuthology.orchestra.run.smithi028.stdout:(21/24): thrift-0.15.0-2.el9.x86_64.rpm 21 MB/s | 1.6 MB 00:00 2024-06-18T02:26:50.619 INFO:teuthology.orchestra.run.smithi028.stdout:(22/24): librbd1-17.2.7-1012.g3fe41055.el9.x86_ 11 MB/s | 2.9 MB 00:00 2024-06-18T02:26:50.852 INFO:teuthology.orchestra.run.smithi028.stdout:(23/24): librados2-17.2.7-1012.g3fe41055.el9.x8 5.6 MB/s | 3.1 MB 00:00 2024-06-18T02:26:51.269 INFO:teuthology.orchestra.run.smithi028.stdout:(24/24): ceph-common-17.2.7-1012.g3fe41055.el9. 7.5 MB/s | 22 MB 00:02 2024-06-18T02:26:51.273 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-06-18T02:26:51.274 INFO:teuthology.orchestra.run.smithi028.stdout:Total 17 MB/s | 58 MB 00:03 2024-06-18T02:26:51.332 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-06-18T02:26:51.368 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-06-18T02:26:51.368 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-06-18T02:26:51.896 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-06-18T02:26:51.896 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-06-18T02:26:53.096 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-06-18T02:26:53.432 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-06-18T02:26:53.530 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : librados2-2:17.2.7-1012.g3fe41055.el9.x86_64 2/26 2024-06-18T02:26:53.623 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librados2-2:17.2.7-1012.g3fe41055.el9.x86_64 2/26 2024-06-18T02:26:53.703 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rados-2:17.2.7-1012.g3fe41055.el9.x86_64 3/26 2024-06-18T02:26:53.731 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libcephfs2-2:17.2.7-1012.g3fe41055.el9.x86_64 4/26 2024-06-18T02:26:53.825 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libcephfs2-2:17.2.7-1012.g3fe41055.el9.x86_64 4/26 2024-06-18T02:26:53.831 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - CRB 4.5 MB/s | 6.3 MB 00:01 2024-06-18T02:26:53.875 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-06-18T02:26:53.920 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-06-18T02:26:54.379 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-06-18T02:26:54.410 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librgw2-2:17.2.7-1012.g3fe41055.el9.x86_64 8/26 2024-06-18T02:26:54.473 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librgw2-2:17.2.7-1012.g3fe41055.el9.x86_64 8/26 2024-06-18T02:26:54.510 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-06-18T02:26:54.553 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-ceph-argparse-2:17.2.7-1012.g3fe41055.el9. 10/26 2024-06-18T02:26:54.589 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cephfs-2:17.2.7-1012.g3fe41055.el9.x86_64 11/26 2024-06-18T02:26:54.653 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rgw-2:17.2.7-1012.g3fe41055.el9.x86_64 12/26 2024-06-18T02:26:54.679 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libradosstriper1-2:17.2.7-1012.g3fe41055.el9.x86_6 13/26 2024-06-18T02:26:54.746 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1012.g3fe41055.el9.x86_6 13/26 2024-06-18T02:26:54.785 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-06-18T02:26:54.830 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-06-18T02:26:54.862 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-06-18T02:26:55.173 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-06-18T02:26:55.208 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : librbd1-2:17.2.7-1012.g3fe41055.el9.x86_64 18/26 2024-06-18T02:26:55.257 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - Extras packages 120 kB/s | 17 kB 00:00 2024-06-18T02:26:55.283 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librbd1-2:17.2.7-1012.g3fe41055.el9.x86_64 18/26 2024-06-18T02:26:55.358 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rbd-2:17.2.7-1012.g3fe41055.el9.x86_64 19/26 2024-06-18T02:26:55.386 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-ceph-common-2:17.2.7-1012.g3fe41055.el9.x8 20/26 2024-06-18T02:26:56.144 INFO:teuthology.orchestra.run.smithi001.stdout:Extra Packages for Enterprise Linux 29 MB/s | 21 MB 00:00 2024-06-18T02:26:57.903 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-common-2:17.2.7-1012.g3fe41055.el9.x86_64 21/26 2024-06-18T02:26:57.941 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-common-2:17.2.7-1012.g3fe41055.el9.x86_64 21/26 2024-06-18T02:26:58.496 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-common-2:17.2.7-1012.g3fe41055.el9.x86_64 21/26 2024-06-18T02:26:58.537 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-base-2:17.2.7-1012.g3fe41055.el9.x86_64 22/26 2024-06-18T02:26:58.638 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-base-2:17.2.7-1012.g3fe41055.el9.x86_64 22/26 2024-06-18T02:26:58.638 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-06-18T02:26:58.638 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-06-18T02:26:58.638 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:26:58.671 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-selinux-2:17.2.7-1012.g3fe41055.el9.x86_64 23/26 2024-06-18T02:27:01.600 INFO:teuthology.orchestra.run.smithi001.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-06-18T02:27:04.087 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-06-18T02:27:04.092 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-06-18T02:27:04.092 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-06-18T02:27:04.093 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-06-18T02:27:04.093 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-06-18T02:27:04.093 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-radosgw x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 14 M 2024-06-18T02:27:04.093 INFO:teuthology.orchestra.run.smithi001.stdout:Upgrading: 2024-06-18T02:27:04.093 INFO:teuthology.orchestra.run.smithi001.stdout: librados2 x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 3.1 M 2024-06-18T02:27:04.093 INFO:teuthology.orchestra.run.smithi001.stdout: librbd1 x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 2.9 M 2024-06-18T02:27:04.093 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-06-18T02:27:04.093 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-base x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 5.0 M 2024-06-18T02:27:04.093 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-common x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 22 M 2024-06-18T02:27:04.093 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-selinux x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 25 k 2024-06-18T02:27:04.093 INFO:teuthology.orchestra.run.smithi001.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-06-18T02:27:04.093 INFO:teuthology.orchestra.run.smithi001.stdout: libcephfs2 x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 637 k 2024-06-18T02:27:04.094 INFO:teuthology.orchestra.run.smithi001.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-06-18T02:27:04.094 INFO:teuthology.orchestra.run.smithi001.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-06-18T02:27:04.094 INFO:teuthology.orchestra.run.smithi001.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-06-18T02:27:04.094 INFO:teuthology.orchestra.run.smithi001.stdout: libradosstriper1 x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 458 k 2024-06-18T02:27:04.094 INFO:teuthology.orchestra.run.smithi001.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-06-18T02:27:04.094 INFO:teuthology.orchestra.run.smithi001.stdout: librgw2 x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 4.9 M 2024-06-18T02:27:04.094 INFO:teuthology.orchestra.run.smithi001.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-06-18T02:27:04.094 INFO:teuthology.orchestra.run.smithi001.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-06-18T02:27:04.094 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ceph-argparse x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 45 k 2024-06-18T02:27:04.094 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ceph-common x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 118 k 2024-06-18T02:27:04.094 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cephfs x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 156 k 2024-06-18T02:27:04.094 INFO:teuthology.orchestra.run.smithi001.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-06-18T02:27:04.094 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rados x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 320 k 2024-06-18T02:27:04.095 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rbd x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 293 k 2024-06-18T02:27:04.095 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rgw x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 99 k 2024-06-18T02:27:04.095 INFO:teuthology.orchestra.run.smithi001.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-06-18T02:27:04.095 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:27:04.095 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-06-18T02:27:04.095 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-06-18T02:27:04.095 INFO:teuthology.orchestra.run.smithi001.stdout:Install 22 Packages 2024-06-18T02:27:04.095 INFO:teuthology.orchestra.run.smithi001.stdout:Upgrade 2 Packages 2024-06-18T02:27:04.095 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:27:04.096 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 58 M 2024-06-18T02:27:04.096 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-06-18T02:27:05.360 INFO:teuthology.orchestra.run.smithi001.stdout:(1/24): ceph-base-17.2.7-1012.g3fe41055.el9.x86 5.0 MB/s | 5.0 MB 00:00 2024-06-18T02:27:05.443 INFO:teuthology.orchestra.run.smithi001.stdout:(2/24): ceph-selinux-17.2.7-1012.g3fe41055.el9. 302 kB/s | 25 kB 00:00 2024-06-18T02:27:05.570 INFO:teuthology.orchestra.run.smithi001.stdout:(3/24): libcephfs2-17.2.7-1012.g3fe41055.el9.x8 5.0 MB/s | 637 kB 00:00 2024-06-18T02:27:05.686 INFO:teuthology.orchestra.run.smithi001.stdout:(4/24): libradosstriper1-17.2.7-1012.g3fe41055. 3.8 MB/s | 458 kB 00:00 2024-06-18T02:27:06.153 INFO:teuthology.orchestra.run.smithi001.stdout:(5/24): ceph-radosgw-17.2.7-1012.g3fe41055.el9. 7.9 MB/s | 14 MB 00:01 2024-06-18T02:27:06.236 INFO:teuthology.orchestra.run.smithi001.stdout:(6/24): python3-ceph-argparse-17.2.7-1012.g3fe4 542 kB/s | 45 kB 00:00 2024-06-18T02:27:06.312 INFO:teuthology.orchestra.run.smithi001.stdout:(7/24): python3-ceph-common-17.2.7-1012.g3fe410 1.5 MB/s | 118 kB 00:00 2024-06-18T02:27:06.420 INFO:teuthology.orchestra.run.smithi001.stdout:(8/24): librgw2-17.2.7-1012.g3fe41055.el9.x86_6 6.7 MB/s | 4.9 MB 00:00 2024-06-18T02:27:06.446 INFO:teuthology.orchestra.run.smithi001.stdout:(9/24): python3-cephfs-17.2.7-1012.g3fe41055.el 1.1 MB/s | 156 kB 00:00 2024-06-18T02:27:06.671 INFO:teuthology.orchestra.run.smithi001.stdout:(10/24): ceph-common-17.2.7-1012.g3fe41055.el9. 9.8 MB/s | 22 MB 00:02 2024-06-18T02:27:06.696 INFO:teuthology.orchestra.run.smithi001.stdout:(11/24): python3-rbd-17.2.7-1012.g3fe41055.el9. 1.1 MB/s | 293 kB 00:00 2024-06-18T02:27:06.722 INFO:teuthology.orchestra.run.smithi001.stdout:(12/24): python3-rados-17.2.7-1012.g3fe41055.el 1.0 MB/s | 320 kB 00:00 2024-06-18T02:27:06.747 INFO:teuthology.orchestra.run.smithi001.stdout:(13/24): python3-rgw-17.2.7-1012.g3fe41055.el9. 1.3 MB/s | 99 kB 00:00 2024-06-18T02:27:07.181 INFO:teuthology.orchestra.run.smithi001.stdout:(14/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 99 kB/s | 45 kB 00:00 2024-06-18T02:27:07.289 INFO:teuthology.orchestra.run.smithi001.stdout:(15/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 270 kB/s | 160 kB 00:00 2024-06-18T02:27:07.406 INFO:teuthology.orchestra.run.smithi001.stdout:(16/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.3 MB/s | 292 kB 00:00 2024-06-18T02:27:07.423 INFO:teuthology.orchestra.run.smithi001.stdout:(17/24): python3-prettytable-0.7.2-27.el9.noarc 317 kB/s | 42 kB 00:00 2024-06-18T02:27:07.515 INFO:teuthology.orchestra.run.smithi001.stdout:(18/24): librdkafka-1.6.1-102.el9.x86_64.rpm 863 kB/s | 662 kB 00:00 2024-06-18T02:27:07.565 INFO:teuthology.orchestra.run.smithi001.stdout:(19/24): liboath-2.6.7-2.el9.x86_64.rpm 342 kB/s | 49 kB 00:00 2024-06-18T02:27:07.590 INFO:teuthology.orchestra.run.smithi001.stdout:(20/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.6 MB/s | 309 kB 00:00 2024-06-18T02:27:07.616 INFO:teuthology.orchestra.run.smithi001.stdout:(21/24): libunwind-1.6.2-1.el9.x86_64.rpm 687 kB/s | 67 kB 00:00 2024-06-18T02:27:07.666 INFO:teuthology.orchestra.run.smithi001.stdout:(22/24): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-06-18T02:27:07.966 INFO:teuthology.orchestra.run.smithi001.stdout:(23/24): librbd1-17.2.7-1012.g3fe41055.el9.x86_ 8.4 MB/s | 2.9 MB 00:00 2024-06-18T02:27:08.025 INFO:teuthology.orchestra.run.smithi001.stdout:(24/24): librados2-17.2.7-1012.g3fe41055.el9.x8 7.1 MB/s | 3.1 MB 00:00 2024-06-18T02:27:08.031 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-06-18T02:27:08.031 INFO:teuthology.orchestra.run.smithi001.stdout:Total 15 MB/s | 58 MB 00:03 2024-06-18T02:27:08.093 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-06-18T02:27:08.128 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-06-18T02:27:08.129 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-06-18T02:27:08.671 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1012.g3fe41055.el9.x86_64 23/26 2024-06-18T02:27:08.671 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /sys 2024-06-18T02:27:08.671 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /proc 2024-06-18T02:27:08.671 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /mnt 2024-06-18T02:27:08.671 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /var/tmp 2024-06-18T02:27:08.671 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /home 2024-06-18T02:27:08.671 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /root 2024-06-18T02:27:08.671 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /tmp 2024-06-18T02:27:08.671 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:27:08.682 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-06-18T02:27:08.682 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-06-18T02:27:09.880 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-06-18T02:27:09.991 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-radosgw-2:17.2.7-1012.g3fe41055.el9.x86_64 24/26 2024-06-18T02:27:10.071 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1012.g3fe41055.el9.x86_64 24/26 2024-06-18T02:27:10.071 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-18T02:27:10.072 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-06-18T02:27:10.072 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-18T02:27:10.072 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-18T02:27:10.072 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:27:10.073 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-06-18T02:27:10.161 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-06-18T02:27:10.161 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-06-18T02:27:10.292 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-06-18T02:27:10.344 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : librados2-2:17.2.7-1012.g3fe41055.el9.x86_64 2/26 2024-06-18T02:27:10.428 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librados2-2:17.2.7-1012.g3fe41055.el9.x86_64 2/26 2024-06-18T02:27:10.509 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-rados-2:17.2.7-1012.g3fe41055.el9.x86_64 3/26 2024-06-18T02:27:10.547 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libcephfs2-2:17.2.7-1012.g3fe41055.el9.x86_64 4/26 2024-06-18T02:27:10.631 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: libcephfs2-2:17.2.7-1012.g3fe41055.el9.x86_64 4/26 2024-06-18T02:27:10.680 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-06-18T02:27:10.717 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-06-18T02:27:11.174 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-06-18T02:27:11.198 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : librgw2-2:17.2.7-1012.g3fe41055.el9.x86_64 8/26 2024-06-18T02:27:11.261 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librgw2-2:17.2.7-1012.g3fe41055.el9.x86_64 8/26 2024-06-18T02:27:11.298 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-06-18T02:27:11.342 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-ceph-argparse-2:17.2.7-1012.g3fe41055.el9. 10/26 2024-06-18T02:27:11.378 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cephfs-2:17.2.7-1012.g3fe41055.el9.x86_64 11/26 2024-06-18T02:27:11.454 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-06-18T02:27:11.454 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-base-2:17.2.7-1012.g3fe41055.el9.x86_64 1/26 2024-06-18T02:27:11.454 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-common-2:17.2.7-1012.g3fe41055.el9.x86_64 2/26 2024-06-18T02:27:11.454 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-radosgw-2:17.2.7-1012.g3fe41055.el9.x86_64 3/26 2024-06-18T02:27:11.454 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-selinux-2:17.2.7-1012.g3fe41055.el9.x86_64 4/26 2024-06-18T02:27:11.455 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libcephfs2-2:17.2.7-1012.g3fe41055.el9.x86_64 5/26 2024-06-18T02:27:11.455 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libradosstriper1-2:17.2.7-1012.g3fe41055.el9.x86_6 6/26 2024-06-18T02:27:11.455 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librgw2-2:17.2.7-1012.g3fe41055.el9.x86_64 7/26 2024-06-18T02:27:11.455 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1012.g3fe41055.el9. 8/26 2024-06-18T02:27:11.455 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-ceph-common-2:17.2.7-1012.g3fe41055.el9.x8 9/26 2024-06-18T02:27:11.455 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cephfs-2:17.2.7-1012.g3fe41055.el9.x86_64 10/26 2024-06-18T02:27:11.455 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rados-2:17.2.7-1012.g3fe41055.el9.x86_64 11/26 2024-06-18T02:27:11.455 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rbd-2:17.2.7-1012.g3fe41055.el9.x86_64 12/26 2024-06-18T02:27:11.455 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rgw-2:17.2.7-1012.g3fe41055.el9.x86_64 13/26 2024-06-18T02:27:11.455 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-06-18T02:27:11.455 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-06-18T02:27:11.455 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-06-18T02:27:11.455 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-06-18T02:27:11.456 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-06-18T02:27:11.456 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-06-18T02:27:11.456 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-06-18T02:27:11.456 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-06-18T02:27:11.456 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-06-18T02:27:11.456 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librados2-2:17.2.7-1012.g3fe41055.el9.x86_64 23/26 2024-06-18T02:27:11.456 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-06-18T02:27:11.456 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librbd1-2:17.2.7-1012.g3fe41055.el9.x86_64 25/26 2024-06-18T02:27:11.459 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-rgw-2:17.2.7-1012.g3fe41055.el9.x86_64 12/26 2024-06-18T02:27:11.484 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libradosstriper1-2:17.2.7-1012.g3fe41055.el9.x86_6 13/26 2024-06-18T02:27:11.534 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1012.g3fe41055.el9.x86_6 13/26 2024-06-18T02:27:11.575 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-06-18T02:27:11.618 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-06-18T02:27:11.651 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-06-18T02:27:11.960 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-06-18T02:27:11.997 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : librbd1-2:17.2.7-1012.g3fe41055.el9.x86_64 18/26 2024-06-18T02:27:12.088 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librbd1-2:17.2.7-1012.g3fe41055.el9.x86_64 18/26 2024-06-18T02:27:12.164 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-rbd-2:17.2.7-1012.g3fe41055.el9.x86_64 19/26 2024-06-18T02:27:12.171 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-06-18T02:27:12.171 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:27:12.171 INFO:teuthology.orchestra.run.smithi028.stdout:Upgraded: 2024-06-18T02:27:12.171 INFO:teuthology.orchestra.run.smithi028.stdout: librados2-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:27:12.171 INFO:teuthology.orchestra.run.smithi028.stdout: librbd1-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:27:12.171 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-06-18T02:27:12.171 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-base-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:27:12.171 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-common-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:27:12.171 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-radosgw-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:27:12.171 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-selinux-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:27:12.171 INFO:teuthology.orchestra.run.smithi028.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-06-18T02:27:12.171 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs2-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:27:12.171 INFO:teuthology.orchestra.run.smithi028.stdout: liboath-2.6.7-2.el9.x86_64 2024-06-18T02:27:12.172 INFO:teuthology.orchestra.run.smithi028.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-06-18T02:27:12.172 INFO:teuthology.orchestra.run.smithi028.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-06-18T02:27:12.172 INFO:teuthology.orchestra.run.smithi028.stdout: libradosstriper1-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:27:12.172 INFO:teuthology.orchestra.run.smithi028.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-06-18T02:27:12.172 INFO:teuthology.orchestra.run.smithi028.stdout: librgw2-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:27:12.172 INFO:teuthology.orchestra.run.smithi028.stdout: libunwind-1.6.2-1.el9.x86_64 2024-06-18T02:27:12.172 INFO:teuthology.orchestra.run.smithi028.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-06-18T02:27:12.172 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-argparse-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:27:12.172 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-common-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:27:12.172 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cephfs-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:27:12.172 INFO:teuthology.orchestra.run.smithi028.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-06-18T02:27:12.172 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rados-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:27:12.172 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rbd-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:27:12.172 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rgw-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:27:12.172 INFO:teuthology.orchestra.run.smithi028.stdout: thrift-0.15.0-2.el9.x86_64 2024-06-18T02:27:12.173 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:27:12.173 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-06-18T02:27:12.191 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-ceph-common-2:17.2.7-1012.g3fe41055.el9.x8 20/26 2024-06-18T02:27:12.522 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-test 2024-06-18T02:27:13.112 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:00:28 ago on Tue 18 Jun 2024 02:26:45 AM UTC. 2024-06-18T02:27:13.243 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-06-18T02:27:13.243 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-06-18T02:27:13.243 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2024-06-18T02:27:13.244 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-06-18T02:27:13.244 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-06-18T02:27:13.244 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-test x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 48 M 2024-06-18T02:27:13.244 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-06-18T02:27:13.244 INFO:teuthology.orchestra.run.smithi028.stdout: libcephsqlite x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 163 k 2024-06-18T02:27:13.244 INFO:teuthology.orchestra.run.smithi028.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-06-18T02:27:13.244 INFO:teuthology.orchestra.run.smithi028.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-06-18T02:27:13.244 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:27:13.244 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-06-18T02:27:13.244 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-06-18T02:27:13.244 INFO:teuthology.orchestra.run.smithi028.stdout:Install 4 Packages 2024-06-18T02:27:13.244 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:27:13.245 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 48 M 2024-06-18T02:27:13.245 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 192 M 2024-06-18T02:27:13.245 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-06-18T02:27:13.690 INFO:teuthology.orchestra.run.smithi028.stdout:(1/4): libcephsqlite-17.2.7-1012.g3fe41055.el9. 404 kB/s | 163 kB 00:00 2024-06-18T02:27:13.907 INFO:teuthology.orchestra.run.smithi028.stdout:(2/4): socat-1.7.4.1-6.el9.x86_64.rpm 490 kB/s | 304 kB 00:00 2024-06-18T02:27:14.032 INFO:teuthology.orchestra.run.smithi028.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 186 kB/s | 64 kB 00:00 2024-06-18T02:27:14.711 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-common-2:17.2.7-1012.g3fe41055.el9.x86_64 21/26 2024-06-18T02:27:14.749 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-common-2:17.2.7-1012.g3fe41055.el9.x86_64 21/26 2024-06-18T02:27:15.321 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-common-2:17.2.7-1012.g3fe41055.el9.x86_64 21/26 2024-06-18T02:27:15.353 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-base-2:17.2.7-1012.g3fe41055.el9.x86_64 22/26 2024-06-18T02:27:15.446 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-base-2:17.2.7-1012.g3fe41055.el9.x86_64 22/26 2024-06-18T02:27:15.446 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-06-18T02:27:15.446 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-06-18T02:27:15.446 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:27:15.479 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-selinux-2:17.2.7-1012.g3fe41055.el9.x86_64 23/26 2024-06-18T02:27:17.307 INFO:teuthology.orchestra.run.smithi028.stdout:(4/4): ceph-test-17.2.7-1012.g3fe41055.el9.x86_ 12 MB/s | 48 MB 00:04 2024-06-18T02:27:17.309 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-06-18T02:27:17.309 INFO:teuthology.orchestra.run.smithi028.stdout:Total 12 MB/s | 48 MB 00:04 2024-06-18T02:27:17.337 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-06-18T02:27:17.359 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-06-18T02:27:17.359 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-06-18T02:27:17.742 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-06-18T02:27:17.743 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-06-18T02:27:18.276 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-06-18T02:27:18.353 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-06-18T02:27:18.398 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-06-18T02:27:18.425 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libcephsqlite-2:17.2.7-1012.g3fe41055.el9.x86_64 3/4 2024-06-18T02:27:23.106 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1012.g3fe41055.el9.x86_64 3/4 2024-06-18T02:27:23.193 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-test-2:17.2.7-1012.g3fe41055.el9.x86_64 4/4 2024-06-18T02:27:25.407 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-test-2:17.2.7-1012.g3fe41055.el9.x86_64 4/4 2024-06-18T02:27:25.408 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-test-2:17.2.7-1012.g3fe41055.el9.x86_64 1/4 2024-06-18T02:27:25.408 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libcephsqlite-2:17.2.7-1012.g3fe41055.el9.x86_64 2/4 2024-06-18T02:27:25.408 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-06-18T02:27:25.471 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1012.g3fe41055.el9.x86_64 23/26 2024-06-18T02:27:25.471 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /sys 2024-06-18T02:27:25.471 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /proc 2024-06-18T02:27:25.471 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /mnt 2024-06-18T02:27:25.471 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /var/tmp 2024-06-18T02:27:25.471 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /home 2024-06-18T02:27:25.471 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /root 2024-06-18T02:27:25.471 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /tmp 2024-06-18T02:27:25.471 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:27:25.690 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-06-18T02:27:25.690 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:27:25.690 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-06-18T02:27:25.690 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-test-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:27:25.690 INFO:teuthology.orchestra.run.smithi028.stdout: libcephsqlite-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:27:25.690 INFO:teuthology.orchestra.run.smithi028.stdout: socat-1.7.4.1-6.el9.x86_64 2024-06-18T02:27:25.690 INFO:teuthology.orchestra.run.smithi028.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-06-18T02:27:25.690 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:27:25.690 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-06-18T02:27:26.024 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph 2024-06-18T02:27:26.618 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:00:41 ago on Tue 18 Jun 2024 02:26:45 AM UTC. 2024-06-18T02:27:26.756 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-06-18T02:27:26.759 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-06-18T02:27:26.759 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2024-06-18T02:27:26.759 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-06-18T02:27:26.759 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-06-18T02:27:26.759 INFO:teuthology.orchestra.run.smithi028.stdout: ceph x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 6.5 k 2024-06-18T02:27:26.759 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-06-18T02:27:26.759 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mds x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 2.1 M 2024-06-18T02:27:26.760 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 1.4 M 2024-06-18T02:27:26.760 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1012.g3fe41055.el9 ceph-noarch 240 k 2024-06-18T02:27:26.760 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mon x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 3.9 M 2024-06-18T02:27:26.760 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-osd x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 15 M 2024-06-18T02:27:26.760 INFO:teuthology.orchestra.run.smithi028.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-06-18T02:27:26.760 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-06-18T02:27:26.760 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-06-18T02:27:26.760 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-06-18T02:27:26.760 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-06-18T02:27:26.760 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-06-18T02:27:26.760 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-06-18T02:27:26.760 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-collections 2024-06-18T02:27:26.760 INFO:teuthology.orchestra.run.smithi028.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-06-18T02:27:26.761 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-06-18T02:27:26.761 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-06-18T02:27:26.761 INFO:teuthology.orchestra.run.smithi028.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-06-18T02:27:26.761 INFO:teuthology.orchestra.run.smithi028.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-06-18T02:27:26.761 INFO:teuthology.orchestra.run.smithi028.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-06-18T02:27:26.761 INFO:teuthology.orchestra.run.smithi028.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-06-18T02:27:26.761 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-06-18T02:27:26.761 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-06-18T02:27:26.761 INFO:teuthology.orchestra.run.smithi028.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-06-18T02:27:26.761 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-06-18T02:27:26.761 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-06-18T02:27:26.761 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-06-18T02:27:26.761 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-06-18T02:27:26.761 INFO:teuthology.orchestra.run.smithi028.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-06-18T02:27:26.762 INFO:teuthology.orchestra.run.smithi028.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-06-18T02:27:26.762 INFO:teuthology.orchestra.run.smithi028.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-06-18T02:27:26.762 INFO:teuthology.orchestra.run.smithi028.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-06-18T02:27:26.762 INFO:teuthology.orchestra.run.smithi028.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-06-18T02:27:26.762 INFO:teuthology.orchestra.run.smithi028.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-06-18T02:27:26.762 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:27:26.762 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-06-18T02:27:26.762 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-06-18T02:27:26.762 INFO:teuthology.orchestra.run.smithi028.stdout:Install 33 Packages 2024-06-18T02:27:26.762 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:27:26.763 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 27 M 2024-06-18T02:27:26.763 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 96 M 2024-06-18T02:27:26.763 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-06-18T02:27:26.790 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-radosgw-2:17.2.7-1012.g3fe41055.el9.x86_64 24/26 2024-06-18T02:27:26.870 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1012.g3fe41055.el9.x86_64 24/26 2024-06-18T02:27:26.870 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-18T02:27:26.870 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-06-18T02:27:26.870 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-18T02:27:26.870 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-18T02:27:26.870 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:27:26.872 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-06-18T02:27:26.951 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-06-18T02:27:26.951 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-06-18T02:27:27.250 INFO:teuthology.orchestra.run.smithi028.stdout:(1/33): ceph-17.2.7-1012.g3fe41055.el9.x86_64.r 30 kB/s | 6.5 kB 00:00 2024-06-18T02:27:27.559 INFO:teuthology.orchestra.run.smithi028.stdout:(2/33): ceph-mds-17.2.7-1012.g3fe41055.el9.x86_ 4.0 MB/s | 2.1 MB 00:00 2024-06-18T02:27:27.592 INFO:teuthology.orchestra.run.smithi028.stdout:(3/33): ceph-mgr-17.2.7-1012.g3fe41055.el9.x86_ 2.5 MB/s | 1.4 MB 00:00 2024-06-18T02:27:27.701 INFO:teuthology.orchestra.run.smithi028.stdout:(4/33): ceph-mgr-modules-core-17.2.7-1012.g3fe4 2.2 MB/s | 240 kB 00:00 2024-06-18T02:27:27.801 INFO:teuthology.orchestra.run.smithi028.stdout:(5/33): ceph-mon-17.2.7-1012.g3fe41055.el9.x86_ 7.1 MB/s | 3.9 MB 00:00 2024-06-18T02:27:27.901 INFO:teuthology.orchestra.run.smithi028.stdout:(6/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.2 MB/s | 253 kB 00:00 2024-06-18T02:27:27.960 INFO:teuthology.orchestra.run.smithi028.stdout:(7/33): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-06-18T02:27:28.019 INFO:teuthology.orchestra.run.smithi028.stdout:(8/33): python3-pycparser-2.20-6.el9.noarch.rpm 2.3 MB/s | 135 kB 00:00 2024-06-18T02:27:28.061 INFO:teuthology.orchestra.run.smithi028.stdout:(9/33): python3-cryptography-36.0.1-4.el9.x86_6 4.7 MB/s | 1.2 MB 00:00 2024-06-18T02:27:28.086 INFO:teuthology.orchestra.run.smithi028.stdout:(10/33): python3-requests-2.25.1-8.el9.noarch.r 1.8 MB/s | 125 kB 00:00 2024-06-18T02:27:28.111 INFO:teuthology.orchestra.run.smithi028.stdout:(11/33): python3-urllib3-1.26.5-5.el9.noarch.rp 4.2 MB/s | 215 kB 00:00 2024-06-18T02:27:28.145 INFO:teuthology.orchestra.run.smithi028.stdout:(12/33): python3-mako-1.1.4-6.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2024-06-18T02:27:28.170 INFO:teuthology.orchestra.run.smithi028.stdout:(13/33): python3-markupsafe-1.1.1-12.el9.x86_64 594 kB/s | 35 kB 00:00 2024-06-18T02:27:28.195 INFO:teuthology.orchestra.run.smithi028.stdout:(14/33): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-06-18T02:27:28.208 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-06-18T02:27:28.208 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-base-2:17.2.7-1012.g3fe41055.el9.x86_64 1/26 2024-06-18T02:27:28.208 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-common-2:17.2.7-1012.g3fe41055.el9.x86_64 2/26 2024-06-18T02:27:28.208 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-radosgw-2:17.2.7-1012.g3fe41055.el9.x86_64 3/26 2024-06-18T02:27:28.208 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-selinux-2:17.2.7-1012.g3fe41055.el9.x86_64 4/26 2024-06-18T02:27:28.208 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libcephfs2-2:17.2.7-1012.g3fe41055.el9.x86_64 5/26 2024-06-18T02:27:28.208 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libradosstriper1-2:17.2.7-1012.g3fe41055.el9.x86_6 6/26 2024-06-18T02:27:28.209 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librgw2-2:17.2.7-1012.g3fe41055.el9.x86_64 7/26 2024-06-18T02:27:28.209 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1012.g3fe41055.el9. 8/26 2024-06-18T02:27:28.209 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-ceph-common-2:17.2.7-1012.g3fe41055.el9.x8 9/26 2024-06-18T02:27:28.209 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cephfs-2:17.2.7-1012.g3fe41055.el9.x86_64 10/26 2024-06-18T02:27:28.209 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-rados-2:17.2.7-1012.g3fe41055.el9.x86_64 11/26 2024-06-18T02:27:28.209 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-rbd-2:17.2.7-1012.g3fe41055.el9.x86_64 12/26 2024-06-18T02:27:28.209 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-rgw-2:17.2.7-1012.g3fe41055.el9.x86_64 13/26 2024-06-18T02:27:28.209 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-06-18T02:27:28.209 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-06-18T02:27:28.209 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-06-18T02:27:28.209 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-06-18T02:27:28.209 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-06-18T02:27:28.209 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-06-18T02:27:28.210 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-06-18T02:27:28.210 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-06-18T02:27:28.210 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-06-18T02:27:28.210 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librados2-2:17.2.7-1012.g3fe41055.el9.x86_64 23/26 2024-06-18T02:27:28.210 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-06-18T02:27:28.210 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librbd1-2:17.2.7-1012.g3fe41055.el9.x86_64 25/26 2024-06-18T02:27:28.221 INFO:teuthology.orchestra.run.smithi028.stdout:(15/33): python3-toml-0.10.2-6.el9.noarch.rpm 826 kB/s | 42 kB 00:00 2024-06-18T02:27:28.305 INFO:teuthology.orchestra.run.smithi028.stdout:(16/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.0 MB/s | 172 kB 00:00 2024-06-18T02:27:28.338 INFO:teuthology.orchestra.run.smithi028.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 305 kB/s | 43 kB 00:00 2024-06-18T02:27:28.363 INFO:teuthology.orchestra.run.smithi028.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 427 kB/s | 11 kB 00:00 2024-06-18T02:27:28.388 INFO:teuthology.orchestra.run.smithi028.stdout:(19/33): python3-jaraco-classes-3.2.1-5.el9.noa 703 kB/s | 18 kB 00:00 2024-06-18T02:27:28.413 INFO:teuthology.orchestra.run.smithi028.stdout:(20/33): python3-cherrypy-18.6.1-2.el9.noarch.r 3.3 MB/s | 358 kB 00:00 2024-06-18T02:27:28.439 INFO:teuthology.orchestra.run.smithi028.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 461 kB/s | 23 kB 00:00 2024-06-18T02:27:28.464 INFO:teuthology.orchestra.run.smithi028.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 386 kB/s | 19 kB 00:00 2024-06-18T02:27:28.489 INFO:teuthology.orchestra.run.smithi028.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 392 kB/s | 20 kB 00:00 2024-06-18T02:27:28.515 INFO:teuthology.orchestra.run.smithi028.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 923 kB/s | 46 kB 00:00 2024-06-18T02:27:28.540 INFO:teuthology.orchestra.run.smithi028.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-06-18T02:27:28.566 INFO:teuthology.orchestra.run.smithi028.stdout:(26/33): python3-portend-3.1.0-2.el9.noarch.rpm 653 kB/s | 16 kB 00:00 2024-06-18T02:27:28.591 INFO:teuthology.orchestra.run.smithi028.stdout:(27/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 3.5 MB/s | 90 kB 00:00 2024-06-18T02:27:28.616 INFO:teuthology.orchestra.run.smithi028.stdout:(28/33): python3-pecan-1.4.2-3.el9.noarch.rpm 2.6 MB/s | 272 kB 00:00 2024-06-18T02:27:28.641 INFO:teuthology.orchestra.run.smithi028.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 711 kB/s | 36 kB 00:00 2024-06-18T02:27:28.667 INFO:teuthology.orchestra.run.smithi028.stdout:(30/33): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-06-18T02:27:28.692 INFO:teuthology.orchestra.run.smithi028.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 809 kB/s | 20 kB 00:00 2024-06-18T02:27:28.867 INFO:teuthology.orchestra.run.smithi028.stdout:(32/33): ceph-osd-17.2.7-1012.g3fe41055.el9.x86 12 MB/s | 15 MB 00:01 2024-06-18T02:27:28.890 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-06-18T02:27:28.890 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:27:28.890 INFO:teuthology.orchestra.run.smithi001.stdout:Upgraded: 2024-06-18T02:27:28.890 INFO:teuthology.orchestra.run.smithi001.stdout: librados2-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:27:28.890 INFO:teuthology.orchestra.run.smithi001.stdout: librbd1-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:27:28.890 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-06-18T02:27:28.890 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-base-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:27:28.891 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-common-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:27:28.891 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-radosgw-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:27:28.891 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-selinux-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:27:28.891 INFO:teuthology.orchestra.run.smithi001.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-06-18T02:27:28.891 INFO:teuthology.orchestra.run.smithi001.stdout: libcephfs2-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:27:28.891 INFO:teuthology.orchestra.run.smithi001.stdout: liboath-2.6.7-2.el9.x86_64 2024-06-18T02:27:28.891 INFO:teuthology.orchestra.run.smithi001.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-06-18T02:27:28.891 INFO:teuthology.orchestra.run.smithi001.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-06-18T02:27:28.891 INFO:teuthology.orchestra.run.smithi001.stdout: libradosstriper1-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:27:28.891 INFO:teuthology.orchestra.run.smithi001.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-06-18T02:27:28.891 INFO:teuthology.orchestra.run.smithi001.stdout: librgw2-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:27:28.891 INFO:teuthology.orchestra.run.smithi001.stdout: libunwind-1.6.2-1.el9.x86_64 2024-06-18T02:27:28.891 INFO:teuthology.orchestra.run.smithi001.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-06-18T02:27:28.891 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ceph-argparse-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:27:28.892 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ceph-common-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:27:28.892 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cephfs-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:27:28.892 INFO:teuthology.orchestra.run.smithi001.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-06-18T02:27:28.892 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rados-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:27:28.892 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rbd-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:27:28.892 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rgw-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:27:28.892 INFO:teuthology.orchestra.run.smithi001.stdout: thrift-0.15.0-2.el9.x86_64 2024-06-18T02:27:28.892 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:27:28.892 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-06-18T02:27:28.892 INFO:teuthology.orchestra.run.smithi028.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.7 MB/s | 427 kB 00:00 2024-06-18T02:27:28.896 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-06-18T02:27:28.896 INFO:teuthology.orchestra.run.smithi028.stdout:Total 13 MB/s | 27 MB 00:02 2024-06-18T02:27:29.014 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-06-18T02:27:29.053 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-06-18T02:27:29.053 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-06-18T02:27:29.235 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-test 2024-06-18T02:27:29.421 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-06-18T02:27:29.421 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-06-18T02:27:29.829 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:00:28 ago on Tue 18 Jun 2024 02:27:01 AM UTC. 2024-06-18T02:27:29.960 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-06-18T02:27:29.960 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-06-18T02:27:29.960 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-06-18T02:27:29.960 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-06-18T02:27:29.961 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-06-18T02:27:29.961 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-test x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 48 M 2024-06-18T02:27:29.961 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-06-18T02:27:29.961 INFO:teuthology.orchestra.run.smithi001.stdout: libcephsqlite x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 163 k 2024-06-18T02:27:29.961 INFO:teuthology.orchestra.run.smithi001.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-06-18T02:27:29.961 INFO:teuthology.orchestra.run.smithi001.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-06-18T02:27:29.961 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:27:29.961 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-06-18T02:27:29.961 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-06-18T02:27:29.961 INFO:teuthology.orchestra.run.smithi001.stdout:Install 4 Packages 2024-06-18T02:27:29.961 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:27:29.962 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 48 M 2024-06-18T02:27:29.962 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 192 M 2024-06-18T02:27:29.962 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-06-18T02:27:30.222 INFO:teuthology.orchestra.run.smithi001.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.4 MB/s | 304 kB 00:00 2024-06-18T02:27:30.247 INFO:teuthology.orchestra.run.smithi001.stdout:(2/4): libcephsqlite-17.2.7-1012.g3fe41055.el9. 667 kB/s | 163 kB 00:00 2024-06-18T02:27:30.364 INFO:teuthology.orchestra.run.smithi001.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 449 kB/s | 64 kB 00:00 2024-06-18T02:27:30.569 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-06-18T02:27:30.637 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-06-18T02:27:30.670 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-06-18T02:27:30.703 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-06-18T02:27:30.800 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-06-18T02:27:30.885 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-06-18T02:27:30.931 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-06-18T02:27:30.966 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-06-18T02:27:30.995 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-06-18T02:27:31.029 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-06-18T02:27:31.063 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-06-18T02:27:31.104 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-06-18T02:27:31.147 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-06-18T02:27:31.181 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-06-18T02:27:31.220 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-06-18T02:27:31.304 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-06-18T02:27:31.458 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-06-18T02:27:31.539 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-06-18T02:27:31.599 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 18/33 2024-06-18T02:27:31.636 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-06-18T02:27:31.695 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-06-18T02:27:31.766 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-06-18T02:27:31.972 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-06-18T02:27:32.023 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-06-18T02:27:32.112 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-06-18T02:27:32.211 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-06-18T02:27:32.247 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-06-18T02:27:32.370 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-06-18T02:27:32.481 INFO:teuthology.orchestra.run.smithi001.stdout:(4/4): ceph-test-17.2.7-1012.g3fe41055.el9.x86_ 19 MB/s | 48 MB 00:02 2024-06-18T02:27:32.483 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-06-18T02:27:32.483 INFO:teuthology.orchestra.run.smithi001.stdout:Total 19 MB/s | 48 MB 00:02 2024-06-18T02:27:32.512 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-06-18T02:27:32.532 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1012.g3fe41055.el9. 28/33 2024-06-18T02:27:32.533 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-06-18T02:27:32.533 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-06-18T02:27:32.565 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-2:17.2.7-1012.g3fe41055.el9.x86_64 29/33 2024-06-18T02:27:32.592 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1012.g3fe41055.el9.x86_64 29/33 2024-06-18T02:27:32.592 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-18T02:27:32.593 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-06-18T02:27:32.593 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-18T02:27:32.593 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-18T02:27:32.593 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:27:32.918 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-06-18T02:27:32.918 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-06-18T02:27:33.487 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-06-18T02:27:33.575 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-06-18T02:27:33.614 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-06-18T02:27:33.649 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libcephsqlite-2:17.2.7-1012.g3fe41055.el9.x86_64 3/4 2024-06-18T02:27:33.983 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-osd-2:17.2.7-1012.g3fe41055.el9.x86_64 30/33 2024-06-18T02:27:34.011 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-osd-2:17.2.7-1012.g3fe41055.el9.x86_64 30/33 2024-06-18T02:27:34.012 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-18T02:27:34.012 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-06-18T02:27:34.012 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-18T02:27:34.012 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-18T02:27:34.012 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:27:34.408 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mon-2:17.2.7-1012.g3fe41055.el9.x86_64 31/33 2024-06-18T02:27:34.434 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mon-2:17.2.7-1012.g3fe41055.el9.x86_64 31/33 2024-06-18T02:27:34.434 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-18T02:27:34.434 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-06-18T02:27:34.434 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-18T02:27:34.434 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-18T02:27:34.434 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:27:34.642 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mds-2:17.2.7-1012.g3fe41055.el9.x86_64 32/33 2024-06-18T02:27:34.667 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mds-2:17.2.7-1012.g3fe41055.el9.x86_64 32/33 2024-06-18T02:27:34.667 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-18T02:27:34.668 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-06-18T02:27:34.668 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-18T02:27:34.668 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-18T02:27:34.668 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:27:34.734 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-2:17.2.7-1012.g3fe41055.el9.x86_64 33/33 2024-06-18T02:27:36.390 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-2:17.2.7-1012.g3fe41055.el9.x86_64 33/33 2024-06-18T02:27:36.391 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-2:17.2.7-1012.g3fe41055.el9.x86_64 1/33 2024-06-18T02:27:36.391 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mds-2:17.2.7-1012.g3fe41055.el9.x86_64 2/33 2024-06-18T02:27:36.391 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-2:17.2.7-1012.g3fe41055.el9.x86_64 3/33 2024-06-18T02:27:36.391 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mon-2:17.2.7-1012.g3fe41055.el9.x86_64 4/33 2024-06-18T02:27:36.391 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-osd-2:17.2.7-1012.g3fe41055.el9.x86_64 5/33 2024-06-18T02:27:36.391 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1012.g3fe41055.el9. 6/33 2024-06-18T02:27:36.391 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-06-18T02:27:36.391 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-06-18T02:27:36.391 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-06-18T02:27:36.391 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-06-18T02:27:36.391 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-06-18T02:27:36.391 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/33 2024-06-18T02:27:36.392 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-06-18T02:27:36.392 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-06-18T02:27:36.392 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-06-18T02:27:36.392 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-06-18T02:27:36.392 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-06-18T02:27:36.392 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-06-18T02:27:36.392 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-06-18T02:27:36.392 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-06-18T02:27:36.392 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-06-18T02:27:36.392 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-06-18T02:27:36.392 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-06-18T02:27:36.392 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-06-18T02:27:36.392 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-06-18T02:27:36.393 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-06-18T02:27:36.393 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-06-18T02:27:36.393 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-06-18T02:27:36.393 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-06-18T02:27:36.394 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-06-18T02:27:36.394 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-06-18T02:27:36.394 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-06-18T02:27:37.249 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-06-18T02:27:37.249 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:27:37.249 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-06-18T02:27:37.249 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:27:37.249 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mds-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:27:37.250 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:27:37.250 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-modules-core-2:17.2.7-1012.g3fe41055.el9.noarch 2024-06-18T02:27:37.250 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mon-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:27:37.250 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-osd-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:27:37.250 INFO:teuthology.orchestra.run.smithi028.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-06-18T02:27:37.250 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-06-18T02:27:37.250 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-06-18T02:27:37.250 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-06-18T02:27:37.250 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-06-18T02:27:37.250 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-06-18T02:27:37.250 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-06-18T02:27:37.250 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-06-18T02:27:37.250 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-06-18T02:27:37.250 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-06-18T02:27:37.251 INFO:teuthology.orchestra.run.smithi028.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-06-18T02:27:37.251 INFO:teuthology.orchestra.run.smithi028.stdout: python3-mako-1.1.4-6.el9.noarch 2024-06-18T02:27:37.251 INFO:teuthology.orchestra.run.smithi028.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-06-18T02:27:37.251 INFO:teuthology.orchestra.run.smithi028.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-06-18T02:27:37.251 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-06-18T02:27:37.251 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ply-3.11-14.el9.noarch 2024-06-18T02:27:37.251 INFO:teuthology.orchestra.run.smithi028.stdout: python3-portend-3.1.0-2.el9.noarch 2024-06-18T02:27:37.251 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-06-18T02:27:37.251 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pycparser-2.20-6.el9.noarch 2024-06-18T02:27:37.251 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pytz-2021.1-5.el9.noarch 2024-06-18T02:27:37.251 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests-2.25.1-8.el9.noarch 2024-06-18T02:27:37.251 INFO:teuthology.orchestra.run.smithi028.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-06-18T02:27:37.251 INFO:teuthology.orchestra.run.smithi028.stdout: python3-toml-0.10.2-6.el9.noarch 2024-06-18T02:27:37.252 INFO:teuthology.orchestra.run.smithi028.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-06-18T02:27:37.252 INFO:teuthology.orchestra.run.smithi028.stdout: python3-webob-1.8.7-6.el9.noarch 2024-06-18T02:27:37.252 INFO:teuthology.orchestra.run.smithi028.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-06-18T02:27:37.252 INFO:teuthology.orchestra.run.smithi028.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-06-18T02:27:37.252 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:27:37.252 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-06-18T02:27:37.651 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-base 2024-06-18T02:27:38.254 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:00:53 ago on Tue 18 Jun 2024 02:26:45 AM UTC. 2024-06-18T02:27:38.312 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1012.g3fe41055.el9.x86_64 3/4 2024-06-18T02:27:38.341 INFO:teuthology.orchestra.run.smithi028.stdout:Package ceph-base-2:17.2.7-1012.g3fe41055.el9.x86_64 is already installed. 2024-06-18T02:27:38.387 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-06-18T02:27:38.388 INFO:teuthology.orchestra.run.smithi028.stdout:Nothing to do. 2024-06-18T02:27:38.388 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-06-18T02:27:38.409 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-test-2:17.2.7-1012.g3fe41055.el9.x86_64 4/4 2024-06-18T02:27:38.456 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install cephadm 2024-06-18T02:27:39.055 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:00:54 ago on Tue 18 Jun 2024 02:26:45 AM UTC. 2024-06-18T02:27:39.186 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-06-18T02:27:39.187 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-06-18T02:27:39.187 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2024-06-18T02:27:39.187 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-06-18T02:27:39.187 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-06-18T02:27:39.188 INFO:teuthology.orchestra.run.smithi028.stdout: cephadm noarch 2:17.2.7-1012.g3fe41055.el9 ceph-noarch 82 k 2024-06-18T02:27:39.188 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:27:39.188 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-06-18T02:27:39.188 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-06-18T02:27:39.188 INFO:teuthology.orchestra.run.smithi028.stdout:Install 1 Package 2024-06-18T02:27:39.188 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:27:39.188 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 82 k 2024-06-18T02:27:39.188 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 372 k 2024-06-18T02:27:39.188 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-06-18T02:27:39.402 INFO:teuthology.orchestra.run.smithi028.stdout:cephadm-17.2.7-1012.g3fe41055.el9.noarch.rpm 383 kB/s | 82 kB 00:00 2024-06-18T02:27:39.403 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-06-18T02:27:39.403 INFO:teuthology.orchestra.run.smithi028.stdout:Total 380 kB/s | 82 kB 00:00 2024-06-18T02:27:39.404 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-06-18T02:27:39.409 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-06-18T02:27:39.409 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-06-18T02:27:39.411 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-06-18T02:27:39.411 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-06-18T02:27:39.482 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-06-18T02:27:39.936 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: cephadm-2:17.2.7-1012.g3fe41055.el9.noarch 1/1 2024-06-18T02:27:40.050 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : cephadm-2:17.2.7-1012.g3fe41055.el9.noarch 1/1 2024-06-18T02:27:40.401 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: cephadm-2:17.2.7-1012.g3fe41055.el9.noarch 1/1 2024-06-18T02:27:40.644 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : cephadm-2:17.2.7-1012.g3fe41055.el9.noarch 1/1 2024-06-18T02:27:40.644 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:27:40.644 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-06-18T02:27:40.644 INFO:teuthology.orchestra.run.smithi028.stdout: cephadm-2:17.2.7-1012.g3fe41055.el9.noarch 2024-06-18T02:27:40.644 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:27:40.644 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-06-18T02:27:40.699 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-test-2:17.2.7-1012.g3fe41055.el9.x86_64 4/4 2024-06-18T02:27:40.699 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-test-2:17.2.7-1012.g3fe41055.el9.x86_64 1/4 2024-06-18T02:27:40.700 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libcephsqlite-2:17.2.7-1012.g3fe41055.el9.x86_64 2/4 2024-06-18T02:27:40.700 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-06-18T02:27:40.856 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-immutable-object-cache 2024-06-18T02:27:40.989 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-06-18T02:27:40.989 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:27:40.990 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-06-18T02:27:40.990 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-test-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:27:40.990 INFO:teuthology.orchestra.run.smithi001.stdout: libcephsqlite-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:27:40.990 INFO:teuthology.orchestra.run.smithi001.stdout: socat-1.7.4.1-6.el9.x86_64 2024-06-18T02:27:40.990 INFO:teuthology.orchestra.run.smithi001.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-06-18T02:27:40.990 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:27:40.990 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-06-18T02:27:41.257 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph 2024-06-18T02:27:41.457 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:00:56 ago on Tue 18 Jun 2024 02:26:45 AM UTC. 2024-06-18T02:27:41.589 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-06-18T02:27:41.589 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-06-18T02:27:41.590 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repo Size 2024-06-18T02:27:41.590 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-06-18T02:27:41.590 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-06-18T02:27:41.590 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 137 k 2024-06-18T02:27:41.590 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:27:41.590 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-06-18T02:27:41.590 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-06-18T02:27:41.590 INFO:teuthology.orchestra.run.smithi028.stdout:Install 1 Package 2024-06-18T02:27:41.590 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:27:41.590 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 137 k 2024-06-18T02:27:41.590 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 379 k 2024-06-18T02:27:41.591 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-06-18T02:27:41.849 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:00:40 ago on Tue 18 Jun 2024 02:27:01 AM UTC. 2024-06-18T02:27:41.908 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-immutable-object-cache-17.2.7-1012.g3fe410 433 kB/s | 137 kB 00:00 2024-06-18T02:27:41.908 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-06-18T02:27:41.908 INFO:teuthology.orchestra.run.smithi028.stdout:Total 431 kB/s | 137 kB 00:00 2024-06-18T02:27:41.909 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-06-18T02:27:41.917 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-06-18T02:27:41.917 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-06-18T02:27:41.977 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-06-18T02:27:41.978 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-06-18T02:27:41.981 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-06-18T02:27:41.984 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-06-18T02:27:41.984 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-06-18T02:27:41.984 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-06-18T02:27:41.984 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-06-18T02:27:41.984 INFO:teuthology.orchestra.run.smithi001.stdout: ceph x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 6.5 k 2024-06-18T02:27:41.984 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-06-18T02:27:41.985 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mds x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 2.1 M 2024-06-18T02:27:41.985 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 1.4 M 2024-06-18T02:27:41.985 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1012.g3fe41055.el9 ceph-noarch 240 k 2024-06-18T02:27:41.985 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mon x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 3.9 M 2024-06-18T02:27:41.985 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-osd x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 15 M 2024-06-18T02:27:41.985 INFO:teuthology.orchestra.run.smithi001.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-06-18T02:27:41.985 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-06-18T02:27:41.985 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-06-18T02:27:41.985 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-06-18T02:27:41.985 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-06-18T02:27:41.985 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-06-18T02:27:41.985 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-06-18T02:27:41.985 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-collections 2024-06-18T02:27:41.986 INFO:teuthology.orchestra.run.smithi001.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-06-18T02:27:41.986 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-06-18T02:27:41.986 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-06-18T02:27:41.986 INFO:teuthology.orchestra.run.smithi001.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-06-18T02:27:41.986 INFO:teuthology.orchestra.run.smithi001.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-06-18T02:27:41.986 INFO:teuthology.orchestra.run.smithi001.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-06-18T02:27:41.986 INFO:teuthology.orchestra.run.smithi001.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-06-18T02:27:41.986 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-06-18T02:27:41.986 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-06-18T02:27:41.986 INFO:teuthology.orchestra.run.smithi001.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-06-18T02:27:41.986 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-06-18T02:27:41.986 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-06-18T02:27:41.986 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-06-18T02:27:41.987 INFO:teuthology.orchestra.run.smithi001.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-06-18T02:27:41.987 INFO:teuthology.orchestra.run.smithi001.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-06-18T02:27:41.987 INFO:teuthology.orchestra.run.smithi001.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-06-18T02:27:41.987 INFO:teuthology.orchestra.run.smithi001.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-06-18T02:27:41.987 INFO:teuthology.orchestra.run.smithi001.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-06-18T02:27:41.987 INFO:teuthology.orchestra.run.smithi001.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-06-18T02:27:41.987 INFO:teuthology.orchestra.run.smithi001.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-06-18T02:27:41.987 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:27:41.987 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-06-18T02:27:41.987 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-06-18T02:27:41.987 INFO:teuthology.orchestra.run.smithi001.stdout:Install 33 Packages 2024-06-18T02:27:41.987 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:27:41.988 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 27 M 2024-06-18T02:27:41.988 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 96 M 2024-06-18T02:27:41.988 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-06-18T02:27:42.133 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-06-18T02:27:42.217 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1012.g3fe41055. 1/1 2024-06-18T02:27:42.243 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1012.g3fe41055. 1/1 2024-06-18T02:27:42.244 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-18T02:27:42.244 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-06-18T02:27:42.244 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:27:42.466 INFO:teuthology.orchestra.run.smithi001.stdout:(1/33): ceph-17.2.7-1012.g3fe41055.el9.x86_64.r 28 kB/s | 6.5 kB 00:00 2024-06-18T02:27:42.600 INFO:teuthology.orchestra.run.smithi001.stdout:(2/33): ceph-mds-17.2.7-1012.g3fe41055.el9.x86_ 5.6 MB/s | 2.1 MB 00:00 2024-06-18T02:27:42.766 INFO:teuthology.orchestra.run.smithi001.stdout:(3/33): ceph-mgr-17.2.7-1012.g3fe41055.el9.x86_ 2.6 MB/s | 1.4 MB 00:00 2024-06-18T02:27:42.858 INFO:teuthology.orchestra.run.smithi001.stdout:(4/33): ceph-mgr-modules-core-17.2.7-1012.g3fe4 2.6 MB/s | 240 kB 00:00 2024-06-18T02:27:43.034 INFO:teuthology.orchestra.run.smithi001.stdout:(5/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.4 MB/s | 253 kB 00:00 2024-06-18T02:27:43.151 INFO:teuthology.orchestra.run.smithi001.stdout:(6/33): python3-cryptography-36.0.1-4.el9.x86_6 11 MB/s | 1.2 MB 00:00 2024-06-18T02:27:43.200 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1012.g3fe41055. 1/1 2024-06-18T02:27:43.200 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:27:43.201 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-06-18T02:27:43.201 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-immutable-object-cache-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:27:43.201 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:27:43.201 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-06-18T02:27:43.201 INFO:teuthology.orchestra.run.smithi001.stdout:(7/33): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-06-18T02:27:43.251 INFO:teuthology.orchestra.run.smithi001.stdout:(8/33): python3-pycparser-2.20-6.el9.noarch.rpm 2.6 MB/s | 135 kB 00:00 2024-06-18T02:27:43.301 INFO:teuthology.orchestra.run.smithi001.stdout:(9/33): python3-requests-2.25.1-8.el9.noarch.rp 2.4 MB/s | 125 kB 00:00 2024-06-18T02:27:43.368 INFO:teuthology.orchestra.run.smithi001.stdout:(10/33): ceph-mon-17.2.7-1012.g3fe41055.el9.x86 4.3 MB/s | 3.9 MB 00:00 2024-06-18T02:27:43.394 INFO:teuthology.orchestra.run.smithi001.stdout:(11/33): python3-urllib3-1.26.5-5.el9.noarch.rp 2.3 MB/s | 215 kB 00:00 2024-06-18T02:27:43.412 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-mgr 2024-06-18T02:27:43.544 INFO:teuthology.orchestra.run.smithi001.stdout:(12/33): ceph-osd-17.2.7-1012.g3fe41055.el9.x86 16 MB/s | 15 MB 00:00 2024-06-18T02:27:43.594 INFO:teuthology.orchestra.run.smithi001.stdout:(13/33): python3-markupsafe-1.1.1-12.el9.x86_64 173 kB/s | 35 kB 00:00 2024-06-18T02:27:43.619 INFO:teuthology.orchestra.run.smithi001.stdout:(14/33): python3-mako-1.1.4-6.el9.noarch.rpm 687 kB/s | 172 kB 00:00 2024-06-18T02:27:43.636 INFO:teuthology.orchestra.run.smithi001.stdout:(15/33): python3-pytz-2021.1-5.el9.noarch.rpm 554 kB/s | 51 kB 00:00 2024-06-18T02:27:43.653 INFO:teuthology.orchestra.run.smithi001.stdout:(16/33): python3-toml-0.10.2-6.el9.noarch.rpm 708 kB/s | 42 kB 00:00 2024-06-18T02:27:43.754 INFO:teuthology.orchestra.run.smithi001.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 324 kB/s | 43 kB 00:00 2024-06-18T02:27:43.771 INFO:teuthology.orchestra.run.smithi001.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 760 kB/s | 11 kB 00:00 2024-06-18T02:27:43.796 INFO:teuthology.orchestra.run.smithi001.stdout:(19/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2024-06-18T02:27:43.821 INFO:teuthology.orchestra.run.smithi001.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 353 kB/s | 18 kB 00:00 2024-06-18T02:27:43.838 INFO:teuthology.orchestra.run.smithi001.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 553 kB/s | 23 kB 00:00 2024-06-18T02:27:43.855 INFO:teuthology.orchestra.run.smithi001.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 577 kB/s | 19 kB 00:00 2024-06-18T02:27:43.872 INFO:teuthology.orchestra.run.smithi001.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 583 kB/s | 20 kB 00:00 2024-06-18T02:27:43.889 INFO:teuthology.orchestra.run.smithi001.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 1.4 MB/s | 46 kB 00:00 2024-06-18T02:27:43.914 INFO:teuthology.orchestra.run.smithi001.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 1.8 MB/s | 79 kB 00:00 2024-06-18T02:27:43.939 INFO:teuthology.orchestra.run.smithi001.stdout:(26/33): python3-cherrypy-18.6.1-2.el9.noarch.r 1.5 MB/s | 358 kB 00:00 2024-06-18T02:27:43.956 INFO:teuthology.orchestra.run.smithi001.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 394 kB/s | 16 kB 00:00 2024-06-18T02:27:43.973 INFO:teuthology.orchestra.run.smithi001.stdout:(28/33): python3-tempora-5.0.0-2.el9.noarch.rpm 2.1 MB/s | 36 kB 00:00 2024-06-18T02:27:44.007 INFO:teuthology.orchestra.run.smithi001.stdout:(29/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.3 MB/s | 90 kB 00:00 2024-06-18T02:27:44.013 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:00:59 ago on Tue 18 Jun 2024 02:26:45 AM UTC. 2024-06-18T02:27:44.032 INFO:teuthology.orchestra.run.smithi001.stdout:(30/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.9 MB/s | 272 kB 00:00 2024-06-18T02:27:44.058 INFO:teuthology.orchestra.run.smithi001.stdout:(31/33): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-06-18T02:27:44.083 INFO:teuthology.orchestra.run.smithi001.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 399 kB/s | 20 kB 00:00 2024-06-18T02:27:44.101 INFO:teuthology.orchestra.run.smithi028.stdout:Package ceph-mgr-2:17.2.7-1012.g3fe41055.el9.x86_64 is already installed. 2024-06-18T02:27:44.108 INFO:teuthology.orchestra.run.smithi001.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.1 MB/s | 427 kB 00:00 2024-06-18T02:27:44.112 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-06-18T02:27:44.112 INFO:teuthology.orchestra.run.smithi001.stdout:Total 13 MB/s | 27 MB 00:02 2024-06-18T02:27:44.146 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-06-18T02:27:44.148 INFO:teuthology.orchestra.run.smithi028.stdout:Nothing to do. 2024-06-18T02:27:44.148 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-06-18T02:27:44.209 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-mgr-dashboard 2024-06-18T02:27:44.227 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-06-18T02:27:44.267 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-06-18T02:27:44.267 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-06-18T02:27:44.665 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-06-18T02:27:44.665 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-06-18T02:27:44.797 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:00:59 ago on Tue 18 Jun 2024 02:26:45 AM UTC. 2024-06-18T02:27:44.929 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-06-18T02:27:44.929 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-06-18T02:27:44.930 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2024-06-18T02:27:44.930 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-06-18T02:27:44.930 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-06-18T02:27:44.930 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1012.g3fe41055.el9 ceph-noarch 3.4 M 2024-06-18T02:27:44.930 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-06-18T02:27:44.930 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1012.g3fe41055.el9 ceph-noarch 24 k 2024-06-18T02:27:44.930 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1012.g3fe41055.el9 ceph-noarch 15 k 2024-06-18T02:27:44.930 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-06-18T02:27:44.930 INFO:teuthology.orchestra.run.smithi028.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-06-18T02:27:44.930 INFO:teuthology.orchestra.run.smithi028.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-06-18T02:27:44.930 INFO:teuthology.orchestra.run.smithi028.stdout:Installing weak dependencies: 2024-06-18T02:27:44.930 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-06-18T02:27:44.930 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:27:44.931 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-06-18T02:27:44.931 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-06-18T02:27:44.931 INFO:teuthology.orchestra.run.smithi028.stdout:Install 7 Packages 2024-06-18T02:27:44.931 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:27:44.931 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 3.7 M 2024-06-18T02:27:44.931 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 80 M 2024-06-18T02:27:44.931 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-06-18T02:27:45.411 INFO:teuthology.orchestra.run.smithi028.stdout:(1/7): ceph-grafana-dashboards-17.2.7-1012.g3fe 92 kB/s | 24 kB 00:00 2024-06-18T02:27:45.436 INFO:teuthology.orchestra.run.smithi028.stdout:(2/7): ceph-prometheus-alerts-17.2.7-1012.g3fe4 50 kB/s | 15 kB 00:00 2024-06-18T02:27:45.519 INFO:teuthology.orchestra.run.smithi028.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 83 kB/s | 9.0 kB 00:00 2024-06-18T02:27:45.545 INFO:teuthology.orchestra.run.smithi028.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 1.3 MB/s | 31 kB 00:00 2024-06-18T02:27:45.570 INFO:teuthology.orchestra.run.smithi028.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 306 kB/s | 41 kB 00:00 2024-06-18T02:27:45.595 INFO:teuthology.orchestra.run.smithi028.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-06-18T02:27:45.770 INFO:teuthology.orchestra.run.smithi028.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1012.g3fe41055 5.4 MB/s | 3.4 MB 00:00 2024-06-18T02:27:45.774 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-06-18T02:27:45.774 INFO:teuthology.orchestra.run.smithi028.stdout:Total 4.3 MB/s | 3.7 MB 00:00 2024-06-18T02:27:45.774 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-06-18T02:27:45.783 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-06-18T02:27:45.783 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-06-18T02:27:45.850 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-06-18T02:27:45.850 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-06-18T02:27:45.943 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-06-18T02:27:45.994 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-06-18T02:27:46.027 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-06-18T02:27:46.060 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-06-18T02:27:46.156 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-06-18T02:27:46.172 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-06-18T02:27:46.233 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-06-18T02:27:46.238 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-06-18T02:27:46.273 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-06-18T02:27:46.279 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-06-18T02:27:46.314 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-06-18T02:27:46.320 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-06-18T02:27:46.343 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-06-18T02:27:46.359 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-06-18T02:27:46.378 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-06-18T02:27:46.389 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1012.g3fe41055.el9.n 5/7 2024-06-18T02:27:46.411 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-06-18T02:27:46.452 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-06-18T02:27:46.495 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-06-18T02:27:46.531 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-06-18T02:27:46.570 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-06-18T02:27:46.647 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-06-18T02:27:46.798 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-06-18T02:27:46.890 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-06-18T02:27:46.940 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 18/33 2024-06-18T02:27:46.978 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-06-18T02:27:47.036 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-06-18T02:27:47.108 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-06-18T02:27:47.305 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-06-18T02:27:47.357 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-06-18T02:27:47.437 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-06-18T02:27:47.448 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1012.g3fe41055.el9. 6/7 2024-06-18T02:27:47.492 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1012.g3fe41055.el9.noarc 7/7 2024-06-18T02:27:47.535 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-06-18T02:27:47.572 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-06-18T02:27:47.696 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-06-18T02:27:47.858 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1012.g3fe41055.el9. 28/33 2024-06-18T02:27:47.882 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-2:17.2.7-1012.g3fe41055.el9.x86_64 29/33 2024-06-18T02:27:47.909 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1012.g3fe41055.el9.x86_64 29/33 2024-06-18T02:27:47.909 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-18T02:27:47.909 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-06-18T02:27:47.909 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-18T02:27:47.910 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-18T02:27:47.910 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:27:48.512 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1012.g3fe41055.el9.noarc 7/7 2024-06-18T02:27:48.512 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1012.g3fe41055.el9. 1/7 2024-06-18T02:27:48.512 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1012.g3fe41055.el9.noarc 2/7 2024-06-18T02:27:48.512 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1012.g3fe41055.el9.n 3/7 2024-06-18T02:27:48.512 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-06-18T02:27:48.512 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-06-18T02:27:48.512 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-06-18T02:27:48.835 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-06-18T02:27:48.835 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:27:48.835 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-06-18T02:27:48.835 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-grafana-dashboards-2:17.2.7-1012.g3fe41055.el9.noarch 2024-06-18T02:27:48.835 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-dashboard-2:17.2.7-1012.g3fe41055.el9.noarch 2024-06-18T02:27:48.835 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-prometheus-alerts-2:17.2.7-1012.g3fe41055.el9.noarch 2024-06-18T02:27:48.835 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-06-18T02:27:48.836 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-06-18T02:27:48.836 INFO:teuthology.orchestra.run.smithi028.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-06-18T02:27:48.836 INFO:teuthology.orchestra.run.smithi028.stdout: python3-routes-2.5.1-5.el9.noarch 2024-06-18T02:27:48.836 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:27:48.836 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-06-18T02:27:49.024 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-mgr-diskprediction-local 2024-06-18T02:27:49.308 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-osd-2:17.2.7-1012.g3fe41055.el9.x86_64 30/33 2024-06-18T02:27:49.336 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-osd-2:17.2.7-1012.g3fe41055.el9.x86_64 30/33 2024-06-18T02:27:49.336 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-18T02:27:49.336 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-06-18T02:27:49.336 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-18T02:27:49.337 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-18T02:27:49.337 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:27:49.614 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:04 ago on Tue 18 Jun 2024 02:26:45 AM UTC. 2024-06-18T02:27:49.734 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mon-2:17.2.7-1012.g3fe41055.el9.x86_64 31/33 2024-06-18T02:27:49.744 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-06-18T02:27:49.745 INFO:teuthology.orchestra.run.smithi028.stdout:======================================================================================= 2024-06-18T02:27:49.746 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2024-06-18T02:27:49.746 INFO:teuthology.orchestra.run.smithi028.stdout:======================================================================================= 2024-06-18T02:27:49.746 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-06-18T02:27:49.746 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1012.g3fe41055.el9 ceph-noarch 7.4 M 2024-06-18T02:27:49.746 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-06-18T02:27:49.746 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-06-18T02:27:49.746 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-06-18T02:27:49.746 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-06-18T02:27:49.746 INFO:teuthology.orchestra.run.smithi028.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-06-18T02:27:49.746 INFO:teuthology.orchestra.run.smithi028.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-06-18T02:27:49.746 INFO:teuthology.orchestra.run.smithi028.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-06-18T02:27:49.747 INFO:teuthology.orchestra.run.smithi028.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-06-18T02:27:49.747 INFO:teuthology.orchestra.run.smithi028.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-06-18T02:27:49.747 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-06-18T02:27:49.747 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-06-18T02:27:49.747 INFO:teuthology.orchestra.run.smithi028.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-06-18T02:27:49.747 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:27:49.747 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-06-18T02:27:49.747 INFO:teuthology.orchestra.run.smithi028.stdout:======================================================================================= 2024-06-18T02:27:49.747 INFO:teuthology.orchestra.run.smithi028.stdout:Install 12 Packages 2024-06-18T02:27:49.747 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:27:49.748 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 38 M 2024-06-18T02:27:49.748 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 209 M 2024-06-18T02:27:49.748 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-06-18T02:27:49.758 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mon-2:17.2.7-1012.g3fe41055.el9.x86_64 31/33 2024-06-18T02:27:49.758 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-18T02:27:49.758 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-06-18T02:27:49.759 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-18T02:27:49.759 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-18T02:27:49.759 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:27:49.976 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mds-2:17.2.7-1012.g3fe41055.el9.x86_64 32/33 2024-06-18T02:27:50.001 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mds-2:17.2.7-1012.g3fe41055.el9.x86_64 32/33 2024-06-18T02:27:50.001 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-18T02:27:50.001 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-06-18T02:27:50.002 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-18T02:27:50.002 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-18T02:27:50.002 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:27:50.068 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-2:17.2.7-1012.g3fe41055.el9.x86_64 33/33 2024-06-18T02:27:50.286 INFO:teuthology.orchestra.run.smithi028.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.4 MB/s | 195 kB 00:00 2024-06-18T02:27:50.337 INFO:teuthology.orchestra.run.smithi028.stdout:(2/12): libgfortran-11.4.1-3.el9.x86_64.rpm 4.2 MB/s | 803 kB 00:00 2024-06-18T02:27:50.420 INFO:teuthology.orchestra.run.smithi028.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 244 kB/s | 33 kB 00:00 2024-06-18T02:27:50.487 INFO:teuthology.orchestra.run.smithi028.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 273 kB/s | 18 kB 00:00 2024-06-18T02:27:50.579 INFO:teuthology.orchestra.run.smithi028.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 384 kB/s | 35 kB 00:00 2024-06-18T02:27:50.746 INFO:teuthology.orchestra.run.smithi028.stdout:(6/12): ceph-mgr-diskprediction-local-17.2.7-10 12 MB/s | 7.4 MB 00:00 2024-06-18T02:27:50.922 INFO:teuthology.orchestra.run.smithi028.stdout:(7/12): python3-devel-3.9.18-3.el9.x86_64.rpm 1.4 MB/s | 244 kB 00:00 2024-06-18T02:27:51.605 INFO:teuthology.orchestra.run.smithi028.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.4 MB/s | 3.0 MB 00:01 2024-06-18T02:27:51.672 INFO:teuthology.orchestra.run.smithi028.stdout:(9/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 4.3 MB/s | 4.7 MB 00:01 2024-06-18T02:27:51.687 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-2:17.2.7-1012.g3fe41055.el9.x86_64 33/33 2024-06-18T02:27:51.688 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-2:17.2.7-1012.g3fe41055.el9.x86_64 1/33 2024-06-18T02:27:51.688 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mds-2:17.2.7-1012.g3fe41055.el9.x86_64 2/33 2024-06-18T02:27:51.688 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-2:17.2.7-1012.g3fe41055.el9.x86_64 3/33 2024-06-18T02:27:51.688 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mon-2:17.2.7-1012.g3fe41055.el9.x86_64 4/33 2024-06-18T02:27:51.688 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-osd-2:17.2.7-1012.g3fe41055.el9.x86_64 5/33 2024-06-18T02:27:51.688 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1012.g3fe41055.el9. 6/33 2024-06-18T02:27:51.688 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-06-18T02:27:51.688 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-06-18T02:27:51.688 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-06-18T02:27:51.688 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-06-18T02:27:51.688 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-06-18T02:27:51.688 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/33 2024-06-18T02:27:51.688 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-06-18T02:27:51.689 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-06-18T02:27:51.689 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-06-18T02:27:51.689 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-06-18T02:27:51.689 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-06-18T02:27:51.689 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-06-18T02:27:51.689 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-06-18T02:27:51.689 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-06-18T02:27:51.689 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-06-18T02:27:51.689 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-06-18T02:27:51.689 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-06-18T02:27:51.689 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-06-18T02:27:51.689 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-06-18T02:27:51.690 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-06-18T02:27:51.690 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-06-18T02:27:51.690 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-06-18T02:27:51.691 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-06-18T02:27:51.691 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-06-18T02:27:51.691 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-06-18T02:27:51.691 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-06-18T02:27:51.730 INFO:teuthology.orchestra.run.smithi028.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 2.0 MB/s | 250 kB 00:00 2024-06-18T02:27:51.922 INFO:teuthology.orchestra.run.smithi028.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 5.1 MB/s | 5.1 MB 00:01 2024-06-18T02:27:52.537 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-06-18T02:27:52.538 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:27:52.538 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-06-18T02:27:52.538 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:27:52.538 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mds-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:27:52.538 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:27:52.538 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-modules-core-2:17.2.7-1012.g3fe41055.el9.noarch 2024-06-18T02:27:52.538 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mon-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:27:52.538 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-osd-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:27:52.538 INFO:teuthology.orchestra.run.smithi001.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-06-18T02:27:52.538 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-06-18T02:27:52.539 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-06-18T02:27:52.539 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-06-18T02:27:52.539 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-06-18T02:27:52.539 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-06-18T02:27:52.539 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-06-18T02:27:52.539 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-06-18T02:27:52.539 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-06-18T02:27:52.539 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-06-18T02:27:52.539 INFO:teuthology.orchestra.run.smithi001.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-06-18T02:27:52.539 INFO:teuthology.orchestra.run.smithi001.stdout: python3-mako-1.1.4-6.el9.noarch 2024-06-18T02:27:52.539 INFO:teuthology.orchestra.run.smithi001.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-06-18T02:27:52.539 INFO:teuthology.orchestra.run.smithi001.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-06-18T02:27:52.539 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-06-18T02:27:52.539 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ply-3.11-14.el9.noarch 2024-06-18T02:27:52.540 INFO:teuthology.orchestra.run.smithi001.stdout: python3-portend-3.1.0-2.el9.noarch 2024-06-18T02:27:52.540 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-06-18T02:27:52.540 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pycparser-2.20-6.el9.noarch 2024-06-18T02:27:52.540 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pytz-2021.1-5.el9.noarch 2024-06-18T02:27:52.540 INFO:teuthology.orchestra.run.smithi001.stdout: python3-requests-2.25.1-8.el9.noarch 2024-06-18T02:27:52.540 INFO:teuthology.orchestra.run.smithi001.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-06-18T02:27:52.540 INFO:teuthology.orchestra.run.smithi001.stdout: python3-toml-0.10.2-6.el9.noarch 2024-06-18T02:27:52.540 INFO:teuthology.orchestra.run.smithi001.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-06-18T02:27:52.540 INFO:teuthology.orchestra.run.smithi001.stdout: python3-webob-1.8.7-6.el9.noarch 2024-06-18T02:27:52.540 INFO:teuthology.orchestra.run.smithi001.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-06-18T02:27:52.540 INFO:teuthology.orchestra.run.smithi001.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-06-18T02:27:52.540 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:27:52.540 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-06-18T02:27:52.902 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-base 2024-06-18T02:27:53.510 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:00:52 ago on Tue 18 Jun 2024 02:27:01 AM UTC. 2024-06-18T02:27:53.597 INFO:teuthology.orchestra.run.smithi001.stdout:Package ceph-base-2:17.2.7-1012.g3fe41055.el9.x86_64 is already installed. 2024-06-18T02:27:53.643 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-06-18T02:27:53.644 INFO:teuthology.orchestra.run.smithi001.stdout:Nothing to do. 2024-06-18T02:27:53.644 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-06-18T02:27:53.714 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install cephadm 2024-06-18T02:27:54.314 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:00:53 ago on Tue 18 Jun 2024 02:27:01 AM UTC. 2024-06-18T02:27:54.445 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-06-18T02:27:54.446 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-06-18T02:27:54.446 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-06-18T02:27:54.446 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-06-18T02:27:54.446 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-06-18T02:27:54.446 INFO:teuthology.orchestra.run.smithi001.stdout: cephadm noarch 2:17.2.7-1012.g3fe41055.el9 ceph-noarch 82 k 2024-06-18T02:27:54.446 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:27:54.446 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-06-18T02:27:54.447 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-06-18T02:27:54.447 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-06-18T02:27:54.447 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:27:54.447 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 82 k 2024-06-18T02:27:54.447 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 372 k 2024-06-18T02:27:54.447 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-06-18T02:27:54.650 INFO:teuthology.orchestra.run.smithi001.stdout:cephadm-17.2.7-1012.g3fe41055.el9.noarch.rpm 405 kB/s | 82 kB 00:00 2024-06-18T02:27:54.650 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-06-18T02:27:54.650 INFO:teuthology.orchestra.run.smithi001.stdout:Total 402 kB/s | 82 kB 00:00 2024-06-18T02:27:54.651 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-06-18T02:27:54.656 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-06-18T02:27:54.656 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-06-18T02:27:54.658 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-06-18T02:27:54.658 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-06-18T02:27:54.714 INFO:teuthology.orchestra.run.smithi028.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 5.3 MB/s | 16 MB 00:03 2024-06-18T02:27:54.715 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-06-18T02:27:54.715 INFO:teuthology.orchestra.run.smithi028.stdout:Total 7.7 MB/s | 38 MB 00:04 2024-06-18T02:27:54.727 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-06-18T02:27:55.084 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-06-18T02:27:55.111 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-06-18T02:27:55.111 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-06-18T02:27:55.185 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: cephadm-2:17.2.7-1012.g3fe41055.el9.noarch 1/1 2024-06-18T02:27:55.272 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : cephadm-2:17.2.7-1012.g3fe41055.el9.noarch 1/1 2024-06-18T02:27:55.623 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: cephadm-2:17.2.7-1012.g3fe41055.el9.noarch 1/1 2024-06-18T02:27:55.637 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-06-18T02:27:55.637 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-06-18T02:27:55.872 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : cephadm-2:17.2.7-1012.g3fe41055.el9.noarch 1/1 2024-06-18T02:27:55.872 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:27:55.872 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-06-18T02:27:55.872 INFO:teuthology.orchestra.run.smithi001.stdout: cephadm-2:17.2.7-1012.g3fe41055.el9.noarch 2024-06-18T02:27:55.872 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:27:55.872 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-06-18T02:27:56.073 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-immutable-object-cache 2024-06-18T02:27:56.433 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-06-18T02:27:56.488 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-06-18T02:27:56.532 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-06-18T02:27:56.636 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-06-18T02:27:56.672 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-06-18T02:27:56.677 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:00:55 ago on Tue 18 Jun 2024 02:27:01 AM UTC. 2024-06-18T02:27:56.810 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-06-18T02:27:56.810 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-06-18T02:27:56.810 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repo Size 2024-06-18T02:27:56.810 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-06-18T02:27:56.810 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-06-18T02:27:56.810 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 137 k 2024-06-18T02:27:56.811 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:27:56.811 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-06-18T02:27:56.811 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-06-18T02:27:56.811 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-06-18T02:27:56.811 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:27:56.811 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 137 k 2024-06-18T02:27:56.811 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 379 k 2024-06-18T02:27:56.811 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-06-18T02:27:56.860 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-06-18T02:27:56.896 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-06-18T02:27:56.983 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-06-18T02:27:57.040 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-immutable-object-cache-17.2.7-1012.g3fe410 603 kB/s | 137 kB 00:00 2024-06-18T02:27:57.040 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-06-18T02:27:57.040 INFO:teuthology.orchestra.run.smithi001.stdout:Total 599 kB/s | 137 kB 00:00 2024-06-18T02:27:57.040 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-06-18T02:27:57.048 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-06-18T02:27:57.049 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-06-18T02:27:57.109 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-06-18T02:27:57.109 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-06-18T02:27:57.267 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-06-18T02:27:57.315 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1012.g3fe41055. 1/1 2024-06-18T02:27:57.341 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1012.g3fe41055. 1/1 2024-06-18T02:27:57.342 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-18T02:27:57.342 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-06-18T02:27:57.342 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:27:57.678 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-06-18T02:27:57.805 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-06-18T02:27:58.280 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1012.g3fe41055. 1/1 2024-06-18T02:27:58.280 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:27:58.280 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-06-18T02:27:58.280 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-immutable-object-cache-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:27:58.280 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:27:58.280 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-06-18T02:27:58.501 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-mgr 2024-06-18T02:27:59.109 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:00:58 ago on Tue 18 Jun 2024 02:27:01 AM UTC. 2024-06-18T02:27:59.132 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-06-18T02:27:59.195 INFO:teuthology.orchestra.run.smithi001.stdout:Package ceph-mgr-2:17.2.7-1012.g3fe41055.el9.x86_64 is already installed. 2024-06-18T02:27:59.241 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-06-18T02:27:59.242 INFO:teuthology.orchestra.run.smithi001.stdout:Nothing to do. 2024-06-18T02:27:59.242 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-06-18T02:27:59.314 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-mgr-dashboard 2024-06-18T02:27:59.902 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:00:58 ago on Tue 18 Jun 2024 02:27:01 AM UTC. 2024-06-18T02:28:00.035 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-06-18T02:28:00.035 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-06-18T02:28:00.036 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-06-18T02:28:00.036 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-06-18T02:28:00.036 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-06-18T02:28:00.036 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1012.g3fe41055.el9 ceph-noarch 3.4 M 2024-06-18T02:28:00.036 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-06-18T02:28:00.036 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1012.g3fe41055.el9 ceph-noarch 24 k 2024-06-18T02:28:00.036 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1012.g3fe41055.el9 ceph-noarch 15 k 2024-06-18T02:28:00.036 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-06-18T02:28:00.036 INFO:teuthology.orchestra.run.smithi001.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-06-18T02:28:00.036 INFO:teuthology.orchestra.run.smithi001.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-06-18T02:28:00.037 INFO:teuthology.orchestra.run.smithi001.stdout:Installing weak dependencies: 2024-06-18T02:28:00.037 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-06-18T02:28:00.037 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:28:00.037 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-06-18T02:28:00.037 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-06-18T02:28:00.037 INFO:teuthology.orchestra.run.smithi001.stdout:Install 7 Packages 2024-06-18T02:28:00.037 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:28:00.038 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 3.7 M 2024-06-18T02:28:00.038 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 80 M 2024-06-18T02:28:00.038 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-06-18T02:28:00.213 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-06-18T02:28:00.235 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1012.g3fe41 12/12 2024-06-18T02:28:00.285 INFO:teuthology.orchestra.run.smithi001.stdout:(1/7): ceph-prometheus-alerts-17.2.7-1012.g3fe4 73 kB/s | 15 kB 00:00 2024-06-18T02:28:00.327 INFO:teuthology.orchestra.run.smithi001.stdout:(2/7): ceph-grafana-dashboards-17.2.7-1012.g3fe 100 kB/s | 24 kB 00:00 2024-06-18T02:28:00.344 INFO:teuthology.orchestra.run.smithi001.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 154 kB/s | 9.0 kB 00:00 2024-06-18T02:28:00.386 INFO:teuthology.orchestra.run.smithi001.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 1.8 MB/s | 31 kB 00:00 2024-06-18T02:28:00.403 INFO:teuthology.orchestra.run.smithi001.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 560 kB/s | 41 kB 00:00 2024-06-18T02:28:00.428 INFO:teuthology.orchestra.run.smithi001.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 4.7 MB/s | 188 kB 00:00 2024-06-18T02:28:00.711 INFO:teuthology.orchestra.run.smithi001.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1012.g3fe41055 5.3 MB/s | 3.4 MB 00:00 2024-06-18T02:28:00.715 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-06-18T02:28:00.715 INFO:teuthology.orchestra.run.smithi001.stdout:Total 5.4 MB/s | 3.7 MB 00:00 2024-06-18T02:28:00.715 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-06-18T02:28:00.724 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-06-18T02:28:00.724 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-06-18T02:28:00.790 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-06-18T02:28:00.791 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-06-18T02:28:01.101 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-06-18T02:28:01.171 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-06-18T02:28:01.206 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-06-18T02:28:01.262 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-06-18T02:28:01.292 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-06-18T02:28:01.321 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1012.g3fe41055.el9.n 5/7 2024-06-18T02:28:02.373 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1012.g3fe41055.el9. 6/7 2024-06-18T02:28:02.407 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1012.g3fe41055.el9.noarc 7/7 2024-06-18T02:28:02.691 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1012.g3fe41 12/12 2024-06-18T02:28:02.692 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1012.g3fe41 1/12 2024-06-18T02:28:02.692 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-06-18T02:28:02.692 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-06-18T02:28:02.692 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-06-18T02:28:02.692 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-06-18T02:28:02.692 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-06-18T02:28:02.692 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-06-18T02:28:02.692 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-06-18T02:28:02.692 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-06-18T02:28:02.692 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-06-18T02:28:02.692 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-06-18T02:28:03.097 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-06-18T02:28:03.097 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:28:03.097 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-06-18T02:28:03.097 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1012.g3fe41055.el9.noarch 2024-06-18T02:28:03.097 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-06-18T02:28:03.097 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-06-18T02:28:03.097 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-06-18T02:28:03.097 INFO:teuthology.orchestra.run.smithi028.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-06-18T02:28:03.097 INFO:teuthology.orchestra.run.smithi028.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-06-18T02:28:03.097 INFO:teuthology.orchestra.run.smithi028.stdout: openblas-0.3.21-2.el9.x86_64 2024-06-18T02:28:03.097 INFO:teuthology.orchestra.run.smithi028.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-06-18T02:28:03.098 INFO:teuthology.orchestra.run.smithi028.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-06-18T02:28:03.098 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-06-18T02:28:03.098 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-06-18T02:28:03.098 INFO:teuthology.orchestra.run.smithi028.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-06-18T02:28:03.098 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:28:03.098 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-06-18T02:28:03.432 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1012.g3fe41055.el9.noarc 7/7 2024-06-18T02:28:03.432 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1012.g3fe41055.el9. 1/7 2024-06-18T02:28:03.432 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1012.g3fe41055.el9.noarc 2/7 2024-06-18T02:28:03.432 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1012.g3fe41055.el9.n 3/7 2024-06-18T02:28:03.433 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-06-18T02:28:03.433 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-06-18T02:28:03.433 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-06-18T02:28:03.457 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-mgr-rook 2024-06-18T02:28:03.771 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-06-18T02:28:03.772 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:28:03.772 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-06-18T02:28:03.772 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-grafana-dashboards-2:17.2.7-1012.g3fe41055.el9.noarch 2024-06-18T02:28:03.772 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-dashboard-2:17.2.7-1012.g3fe41055.el9.noarch 2024-06-18T02:28:03.772 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-prometheus-alerts-2:17.2.7-1012.g3fe41055.el9.noarch 2024-06-18T02:28:03.772 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-06-18T02:28:03.772 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-06-18T02:28:03.772 INFO:teuthology.orchestra.run.smithi001.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-06-18T02:28:03.772 INFO:teuthology.orchestra.run.smithi001.stdout: python3-routes-2.5.1-5.el9.noarch 2024-06-18T02:28:03.772 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:28:03.772 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-06-18T02:28:03.949 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-mgr-diskprediction-local 2024-06-18T02:28:04.061 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:19 ago on Tue 18 Jun 2024 02:26:45 AM UTC. 2024-06-18T02:28:04.192 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-06-18T02:28:04.193 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-06-18T02:28:04.193 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2024-06-18T02:28:04.193 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-06-18T02:28:04.193 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-06-18T02:28:04.193 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-rook noarch 2:17.2.7-1012.g3fe41055.el9 ceph-noarch 47 k 2024-06-18T02:28:04.194 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-06-18T02:28:04.194 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-06-18T02:28:04.194 INFO:teuthology.orchestra.run.smithi028.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-06-18T02:28:04.194 INFO:teuthology.orchestra.run.smithi028.stdout: python3-google-auth noarch 1:2.30.0-1.el9 epel 213 k 2024-06-18T02:28:04.194 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-06-18T02:28:04.194 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-06-18T02:28:04.194 INFO:teuthology.orchestra.run.smithi028.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-06-18T02:28:04.194 INFO:teuthology.orchestra.run.smithi028.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-06-18T02:28:04.194 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-06-18T02:28:04.194 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-06-18T02:28:04.194 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-06-18T02:28:04.194 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-06-18T02:28:04.195 INFO:teuthology.orchestra.run.smithi028.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-06-18T02:28:04.195 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:28:04.195 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-06-18T02:28:04.195 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-06-18T02:28:04.195 INFO:teuthology.orchestra.run.smithi028.stdout:Install 13 Packages 2024-06-18T02:28:04.195 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:28:04.196 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 2.2 M 2024-06-18T02:28:04.196 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 26 M 2024-06-18T02:28:04.196 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-06-18T02:28:04.423 INFO:teuthology.orchestra.run.smithi028.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 184 kB/s | 19 kB 00:00 2024-06-18T02:28:04.439 INFO:teuthology.orchestra.run.smithi028.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 212 kB/s | 26 kB 00:00 2024-06-18T02:28:04.541 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:03 ago on Tue 18 Jun 2024 02:27:01 AM UTC. 2024-06-18T02:28:04.556 INFO:teuthology.orchestra.run.smithi028.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.6 MB/s | 222 kB 00:00 2024-06-18T02:28:04.591 INFO:teuthology.orchestra.run.smithi028.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.0 MB/s | 159 kB 00:00 2024-06-18T02:28:04.616 INFO:teuthology.orchestra.run.smithi028.stdout:(5/13): ceph-mgr-rook-17.2.7-1012.g3fe41055.el9 159 kB/s | 47 kB 00:00 2024-06-18T02:28:04.642 INFO:teuthology.orchestra.run.smithi028.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-06-18T02:28:04.671 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-06-18T02:28:04.672 INFO:teuthology.orchestra.run.smithi001.stdout:======================================================================================= 2024-06-18T02:28:04.673 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-06-18T02:28:04.673 INFO:teuthology.orchestra.run.smithi001.stdout:======================================================================================= 2024-06-18T02:28:04.673 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-06-18T02:28:04.673 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1012.g3fe41055.el9 ceph-noarch 7.4 M 2024-06-18T02:28:04.673 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-06-18T02:28:04.673 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-06-18T02:28:04.673 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-06-18T02:28:04.673 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-06-18T02:28:04.673 INFO:teuthology.orchestra.run.smithi001.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-06-18T02:28:04.673 INFO:teuthology.orchestra.run.smithi001.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-06-18T02:28:04.673 INFO:teuthology.orchestra.run.smithi001.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-06-18T02:28:04.673 INFO:teuthology.orchestra.run.smithi001.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-06-18T02:28:04.673 INFO:teuthology.orchestra.run.smithi001.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-06-18T02:28:04.674 INFO:teuthology.orchestra.run.smithi001.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-06-18T02:28:04.674 INFO:teuthology.orchestra.run.smithi001.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-06-18T02:28:04.674 INFO:teuthology.orchestra.run.smithi001.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-06-18T02:28:04.674 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:28:04.674 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-06-18T02:28:04.674 INFO:teuthology.orchestra.run.smithi001.stdout:======================================================================================= 2024-06-18T02:28:04.674 INFO:teuthology.orchestra.run.smithi001.stdout:Install 12 Packages 2024-06-18T02:28:04.674 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:28:04.675 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 38 M 2024-06-18T02:28:04.675 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 209 M 2024-06-18T02:28:04.675 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-06-18T02:28:04.692 INFO:teuthology.orchestra.run.smithi028.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.0 MB/s | 279 kB 00:00 2024-06-18T02:28:04.742 INFO:teuthology.orchestra.run.smithi028.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 256 kB/s | 32 kB 00:00 2024-06-18T02:28:04.793 INFO:teuthology.orchestra.run.smithi028.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 119 kB/s | 15 kB 00:00 2024-06-18T02:28:04.818 INFO:teuthology.orchestra.run.smithi028.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.6 MB/s | 59 kB 00:00 2024-06-18T02:28:04.843 INFO:teuthology.orchestra.run.smithi028.stdout:(11/13): python3-google-auth-2.30.0-1.el9.noarc 1.7 MB/s | 213 kB 00:00 2024-06-18T02:28:04.893 INFO:teuthology.orchestra.run.smithi028.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.2 MB/s | 90 kB 00:00 2024-06-18T02:28:04.927 INFO:teuthology.orchestra.run.smithi028.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.5 MB/s | 1.0 MB 00:00 2024-06-18T02:28:04.929 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-06-18T02:28:04.929 INFO:teuthology.orchestra.run.smithi028.stdout:Total 3.0 MB/s | 2.2 MB 00:00 2024-06-18T02:28:04.986 INFO:teuthology.orchestra.run.smithi001.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 195 kB 00:00 2024-06-18T02:28:04.994 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-06-18T02:28:05.010 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-06-18T02:28:05.010 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-06-18T02:28:05.019 INFO:teuthology.orchestra.run.smithi001.stdout:(2/12): libgfortran-11.4.1-3.el9.x86_64.rpm 4.3 MB/s | 803 kB 00:00 2024-06-18T02:28:05.095 INFO:teuthology.orchestra.run.smithi001.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 301 kB/s | 33 kB 00:00 2024-06-18T02:28:05.103 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-06-18T02:28:05.103 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-06-18T02:28:05.137 INFO:teuthology.orchestra.run.smithi001.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 436 kB/s | 18 kB 00:00 2024-06-18T02:28:05.221 INFO:teuthology.orchestra.run.smithi001.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 422 kB/s | 35 kB 00:00 2024-06-18T02:28:05.429 INFO:teuthology.orchestra.run.smithi001.stdout:(6/12): ceph-mgr-diskprediction-local-17.2.7-10 12 MB/s | 7.4 MB 00:00 2024-06-18T02:28:05.604 INFO:teuthology.orchestra.run.smithi001.stdout:(7/12): python3-devel-3.9.18-3.el9.x86_64.rpm 1.4 MB/s | 244 kB 00:00 2024-06-18T02:28:05.746 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-06-18T02:28:05.830 INFO:teuthology.orchestra.run.smithi001.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 3.7 MB/s | 3.0 MB 00:00 2024-06-18T02:28:05.913 INFO:teuthology.orchestra.run.smithi001.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.9 MB/s | 250 kB 00:00 2024-06-18T02:28:05.914 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-06-18T02:28:05.980 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-06-18T02:28:06.044 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-06-18T02:28:06.081 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-06-18T02:28:06.121 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-06-18T02:28:06.231 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-06-18T02:28:06.337 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-google-auth-1:2.30.0-1.el9.noarch 7/13 2024-06-18T02:28:06.389 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-06-18T02:28:06.530 INFO:teuthology.orchestra.run.smithi001.stdout:(10/12): openblas-openmp-0.3.21-2.el9.x86_64.rp 3.6 MB/s | 4.7 MB 00:01 2024-06-18T02:28:06.672 INFO:teuthology.orchestra.run.smithi001.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.8 MB/s | 5.1 MB 00:01 2024-06-18T02:28:07.147 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-06-18T02:28:07.203 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-06-18T02:28:07.236 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-06-18T02:28:07.282 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-06-18T02:28:07.305 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-rook-2:17.2.7-1012.g3fe41055.el9.noarch 13/13 2024-06-18T02:28:08.035 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1012.g3fe41055.el9.noarch 13/13 2024-06-18T02:28:08.035 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1012.g3fe41055.el9.noarch 1/13 2024-06-18T02:28:08.035 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-06-18T02:28:08.035 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-06-18T02:28:08.035 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-06-18T02:28:08.035 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-06-18T02:28:08.035 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-06-18T02:28:08.035 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-06-18T02:28:08.035 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-06-18T02:28:08.036 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-06-18T02:28:08.036 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-google-auth-1:2.30.0-1.el9.noarch 10/13 2024-06-18T02:28:08.036 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-06-18T02:28:08.036 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-06-18T02:28:08.358 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-06-18T02:28:08.358 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:28:08.359 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-06-18T02:28:08.359 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-rook-2:17.2.7-1012.g3fe41055.el9.noarch 2024-06-18T02:28:08.359 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-06-18T02:28:08.359 INFO:teuthology.orchestra.run.smithi028.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-06-18T02:28:08.359 INFO:teuthology.orchestra.run.smithi028.stdout: python3-google-auth-1:2.30.0-1.el9.noarch 2024-06-18T02:28:08.359 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-06-18T02:28:08.359 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-06-18T02:28:08.359 INFO:teuthology.orchestra.run.smithi028.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-06-18T02:28:08.359 INFO:teuthology.orchestra.run.smithi028.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-06-18T02:28:08.359 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-06-18T02:28:08.359 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-06-18T02:28:08.359 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-06-18T02:28:08.359 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rsa-4.9-2.el9.noarch 2024-06-18T02:28:08.359 INFO:teuthology.orchestra.run.smithi028.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-06-18T02:28:08.360 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:28:08.360 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-06-18T02:28:08.609 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-mgr-cephadm 2024-06-18T02:28:08.664 INFO:teuthology.orchestra.run.smithi001.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 5.9 MB/s | 16 MB 00:02 2024-06-18T02:28:08.665 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-06-18T02:28:08.665 INFO:teuthology.orchestra.run.smithi001.stdout:Total 9.5 MB/s | 38 MB 00:03 2024-06-18T02:28:09.034 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-06-18T02:28:09.062 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-06-18T02:28:09.062 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-06-18T02:28:09.207 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:24 ago on Tue 18 Jun 2024 02:26:45 AM UTC. 2024-06-18T02:28:09.340 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-06-18T02:28:09.340 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-06-18T02:28:09.341 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2024-06-18T02:28:09.341 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-06-18T02:28:09.341 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-06-18T02:28:09.341 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1012.g3fe41055.el9 ceph-noarch 123 k 2024-06-18T02:28:09.341 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-06-18T02:28:09.341 INFO:teuthology.orchestra.run.smithi028.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-06-18T02:28:09.341 INFO:teuthology.orchestra.run.smithi028.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-06-18T02:28:09.341 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-06-18T02:28:09.341 INFO:teuthology.orchestra.run.smithi028.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-06-18T02:28:09.341 INFO:teuthology.orchestra.run.smithi028.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-06-18T02:28:09.341 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:28:09.341 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-06-18T02:28:09.342 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-06-18T02:28:09.342 INFO:teuthology.orchestra.run.smithi028.stdout:Install 6 Packages 2024-06-18T02:28:09.342 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:28:09.342 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 7.0 M 2024-06-18T02:28:09.342 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 33 M 2024-06-18T02:28:09.342 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-06-18T02:28:09.584 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-06-18T02:28:09.584 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-06-18T02:28:09.969 INFO:teuthology.orchestra.run.smithi028.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.2 MB/s | 248 kB 00:00 2024-06-18T02:28:10.045 INFO:teuthology.orchestra.run.smithi028.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 7.1 MB/s | 548 kB 00:00 2024-06-18T02:28:10.070 INFO:teuthology.orchestra.run.smithi028.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.3 MB/s | 58 kB 00:00 2024-06-18T02:28:10.095 INFO:teuthology.orchestra.run.smithi028.stdout:(4/6): ceph-mgr-cephadm-17.2.7-1012.g3fe41055.e 383 kB/s | 123 kB 00:00 2024-06-18T02:28:10.120 INFO:teuthology.orchestra.run.smithi028.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 1.0 MB/s | 51 kB 00:00 2024-06-18T02:28:10.362 INFO:teuthology.orchestra.run.smithi028.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 10 MB/s | 6.0 MB 00:00 2024-06-18T02:28:10.364 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-06-18T02:28:10.364 INFO:teuthology.orchestra.run.smithi028.stdout:Total 6.8 MB/s | 7.0 MB 00:01 2024-06-18T02:28:10.380 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-06-18T02:28:10.437 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-06-18T02:28:10.438 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-06-18T02:28:10.450 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-06-18T02:28:10.450 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-06-18T02:28:10.480 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-06-18T02:28:10.555 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-06-18T02:28:10.555 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-06-18T02:28:10.586 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-06-18T02:28:10.622 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-06-18T02:28:10.810 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-06-18T02:28:10.845 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-06-18T02:28:10.887 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-06-18T02:28:10.933 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-06-18T02:28:11.063 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-06-18T02:28:11.115 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-06-18T02:28:11.653 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-06-18T02:28:11.673 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-06-18T02:28:11.770 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-06-18T02:28:11.772 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-06-18T02:28:11.832 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-06-18T02:28:11.859 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1012.g3fe41055.el9.noarch 6/6 2024-06-18T02:28:12.634 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1012.g3fe41055.el9.noarch 6/6 2024-06-18T02:28:12.634 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1012.g3fe41055.el9.noarch 1/6 2024-06-18T02:28:12.634 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-06-18T02:28:12.634 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-06-18T02:28:12.634 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-06-18T02:28:12.634 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-06-18T02:28:12.958 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-06-18T02:28:12.958 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:28:12.958 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-06-18T02:28:12.958 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-cephadm-2:17.2.7-1012.g3fe41055.el9.noarch 2024-06-18T02:28:12.958 INFO:teuthology.orchestra.run.smithi028.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-06-18T02:28:12.958 INFO:teuthology.orchestra.run.smithi028.stdout: python3-babel-2.9.1-2.el9.noarch 2024-06-18T02:28:12.958 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-06-18T02:28:12.958 INFO:teuthology.orchestra.run.smithi028.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-06-18T02:28:12.958 INFO:teuthology.orchestra.run.smithi028.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-06-18T02:28:12.958 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:28:12.959 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-06-18T02:28:13.129 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-06-18T02:28:13.195 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-fuse 2024-06-18T02:28:13.790 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:28 ago on Tue 18 Jun 2024 02:26:45 AM UTC. 2024-06-18T02:28:13.921 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-06-18T02:28:13.922 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-06-18T02:28:13.922 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2024-06-18T02:28:13.922 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-06-18T02:28:13.922 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-06-18T02:28:13.922 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-fuse x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 812 k 2024-06-18T02:28:13.922 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-06-18T02:28:13.922 INFO:teuthology.orchestra.run.smithi028.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-06-18T02:28:13.922 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:28:13.922 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-06-18T02:28:13.923 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-06-18T02:28:13.923 INFO:teuthology.orchestra.run.smithi028.stdout:Install 2 Packages 2024-06-18T02:28:13.923 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:28:13.923 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 892 k 2024-06-18T02:28:13.923 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 2.6 M 2024-06-18T02:28:13.923 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-06-18T02:28:14.215 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-06-18T02:28:14.244 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1012.g3fe41 12/12 2024-06-18T02:28:14.271 INFO:teuthology.orchestra.run.smithi028.stdout:(1/2): ceph-fuse-17.2.7-1012.g3fe41055.el9.x86_ 2.6 MB/s | 812 kB 00:00 2024-06-18T02:28:14.405 INFO:teuthology.orchestra.run.smithi028.stdout:(2/2): fuse-2.9.9-15.el9.x86_64.rpm 183 kB/s | 80 kB 00:00 2024-06-18T02:28:14.405 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-06-18T02:28:14.406 INFO:teuthology.orchestra.run.smithi028.stdout:Total 1.8 MB/s | 892 kB 00:00 2024-06-18T02:28:14.418 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-06-18T02:28:14.441 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-06-18T02:28:14.441 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-06-18T02:28:14.523 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-06-18T02:28:14.523 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-06-18T02:28:14.708 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-06-18T02:28:14.838 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-06-18T02:28:14.906 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-fuse-2:17.2.7-1012.g3fe41055.el9.x86_64 2/2 2024-06-18T02:28:15.551 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1012.g3fe41055.el9.x86_64 2/2 2024-06-18T02:28:15.551 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-fuse-2:17.2.7-1012.g3fe41055.el9.x86_64 1/2 2024-06-18T02:28:15.824 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-06-18T02:28:15.824 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:28:15.824 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-06-18T02:28:15.824 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-fuse-2:17.2.7-1012.g3fe41055.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-06-18T02:28:15.824 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:28:15.825 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-06-18T02:28:16.067 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install librados-devel 2024-06-18T02:28:16.667 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:31 ago on Tue 18 Jun 2024 02:26:45 AM UTC. 2024-06-18T02:28:16.769 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1012.g3fe41 12/12 2024-06-18T02:28:16.769 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1012.g3fe41 1/12 2024-06-18T02:28:16.769 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-06-18T02:28:16.770 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-06-18T02:28:16.770 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-06-18T02:28:16.770 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-06-18T02:28:16.770 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-06-18T02:28:16.770 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-06-18T02:28:16.770 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-06-18T02:28:16.770 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-06-18T02:28:16.770 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-06-18T02:28:16.770 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-06-18T02:28:16.801 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-06-18T02:28:16.801 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-06-18T02:28:16.801 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repo Size 2024-06-18T02:28:16.801 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-06-18T02:28:16.801 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-06-18T02:28:16.801 INFO:teuthology.orchestra.run.smithi028.stdout: librados-devel x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 126 k 2024-06-18T02:28:16.801 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:28:16.802 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-06-18T02:28:16.802 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-06-18T02:28:16.802 INFO:teuthology.orchestra.run.smithi028.stdout:Install 1 Package 2024-06-18T02:28:16.802 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:28:16.802 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 126 k 2024-06-18T02:28:16.802 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 448 k 2024-06-18T02:28:16.803 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-06-18T02:28:17.175 INFO:teuthology.orchestra.run.smithi028.stdout:librados-devel-17.2.7-1012.g3fe41055.el9.x86_64 339 kB/s | 126 kB 00:00 2024-06-18T02:28:17.176 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-06-18T02:28:17.176 INFO:teuthology.orchestra.run.smithi028.stdout:Total 338 kB/s | 126 kB 00:00 2024-06-18T02:28:17.176 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-06-18T02:28:17.182 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-06-18T02:28:17.182 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-06-18T02:28:17.206 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-06-18T02:28:17.206 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:28:17.206 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-06-18T02:28:17.206 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1012.g3fe41055.el9.noarch 2024-06-18T02:28:17.206 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-06-18T02:28:17.207 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-06-18T02:28:17.207 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-06-18T02:28:17.207 INFO:teuthology.orchestra.run.smithi001.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-06-18T02:28:17.207 INFO:teuthology.orchestra.run.smithi001.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-06-18T02:28:17.207 INFO:teuthology.orchestra.run.smithi001.stdout: openblas-0.3.21-2.el9.x86_64 2024-06-18T02:28:17.207 INFO:teuthology.orchestra.run.smithi001.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-06-18T02:28:17.207 INFO:teuthology.orchestra.run.smithi001.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-06-18T02:28:17.207 INFO:teuthology.orchestra.run.smithi001.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-06-18T02:28:17.207 INFO:teuthology.orchestra.run.smithi001.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-06-18T02:28:17.207 INFO:teuthology.orchestra.run.smithi001.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-06-18T02:28:17.207 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:28:17.207 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-06-18T02:28:17.250 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-06-18T02:28:17.250 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-06-18T02:28:17.409 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-06-18T02:28:17.509 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librados-devel-2:17.2.7-1012.g3fe41055.el9.x86_64 1/1 2024-06-18T02:28:17.605 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-mgr-rook 2024-06-18T02:28:17.930 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librados-devel-2:17.2.7-1012.g3fe41055.el9.x86_64 1/1 2024-06-18T02:28:18.189 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librados-devel-2:17.2.7-1012.g3fe41055.el9.x86_64 1/1 2024-06-18T02:28:18.190 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:28:18.190 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-06-18T02:28:18.190 INFO:teuthology.orchestra.run.smithi028.stdout: librados-devel-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:28:18.190 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:28:18.190 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-06-18T02:28:18.204 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:17 ago on Tue 18 Jun 2024 02:27:01 AM UTC. 2024-06-18T02:28:18.336 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-06-18T02:28:18.337 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-06-18T02:28:18.337 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-06-18T02:28:18.337 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-06-18T02:28:18.337 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-06-18T02:28:18.337 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-rook noarch 2:17.2.7-1012.g3fe41055.el9 ceph-noarch 47 k 2024-06-18T02:28:18.337 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-06-18T02:28:18.337 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-06-18T02:28:18.337 INFO:teuthology.orchestra.run.smithi001.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-06-18T02:28:18.337 INFO:teuthology.orchestra.run.smithi001.stdout: python3-google-auth noarch 1:2.30.0-1.el9 epel 213 k 2024-06-18T02:28:18.338 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-06-18T02:28:18.338 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-06-18T02:28:18.338 INFO:teuthology.orchestra.run.smithi001.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-06-18T02:28:18.338 INFO:teuthology.orchestra.run.smithi001.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-06-18T02:28:18.338 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-06-18T02:28:18.338 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-06-18T02:28:18.338 INFO:teuthology.orchestra.run.smithi001.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-06-18T02:28:18.338 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-06-18T02:28:18.338 INFO:teuthology.orchestra.run.smithi001.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-06-18T02:28:18.338 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:28:18.338 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-06-18T02:28:18.338 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-06-18T02:28:18.339 INFO:teuthology.orchestra.run.smithi001.stdout:Install 13 Packages 2024-06-18T02:28:18.339 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:28:18.339 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 2.2 M 2024-06-18T02:28:18.339 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 26 M 2024-06-18T02:28:18.339 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-06-18T02:28:18.386 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install libcephfs2 2024-06-18T02:28:18.688 INFO:teuthology.orchestra.run.smithi001.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 148 kB/s | 19 kB 00:00 2024-06-18T02:28:18.704 INFO:teuthology.orchestra.run.smithi001.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 175 kB/s | 26 kB 00:00 2024-06-18T02:28:18.738 INFO:teuthology.orchestra.run.smithi001.stdout:(3/13): ceph-mgr-rook-17.2.7-1012.g3fe41055.el9 264 kB/s | 47 kB 00:00 2024-06-18T02:28:18.796 INFO:teuthology.orchestra.run.smithi001.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.0 MB/s | 222 kB 00:00 2024-06-18T02:28:18.822 INFO:teuthology.orchestra.run.smithi001.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 159 kB 00:00 2024-06-18T02:28:18.863 INFO:teuthology.orchestra.run.smithi001.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 802 kB/s | 54 kB 00:00 2024-06-18T02:28:18.913 INFO:teuthology.orchestra.run.smithi001.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 352 kB/s | 32 kB 00:00 2024-06-18T02:28:18.930 INFO:teuthology.orchestra.run.smithi001.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 234 kB/s | 15 kB 00:00 2024-06-18T02:28:18.956 INFO:teuthology.orchestra.run.smithi001.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.3 MB/s | 279 kB 00:00 2024-06-18T02:28:18.987 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:33 ago on Tue 18 Jun 2024 02:26:45 AM UTC. 2024-06-18T02:28:19.006 INFO:teuthology.orchestra.run.smithi001.stdout:(10/13): python3-google-auth-2.30.0-1.el9.noarc 2.3 MB/s | 213 kB 00:00 2024-06-18T02:28:19.031 INFO:teuthology.orchestra.run.smithi001.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.9 MB/s | 90 kB 00:00 2024-06-18T02:28:19.048 INFO:teuthology.orchestra.run.smithi001.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 641 kB/s | 59 kB 00:00 2024-06-18T02:28:19.074 INFO:teuthology.orchestra.run.smithi028.stdout:Package libcephfs2-2:17.2.7-1012.g3fe41055.el9.x86_64 is already installed. 2024-06-18T02:28:19.080 INFO:teuthology.orchestra.run.smithi001.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.8 MB/s | 1.0 MB 00:00 2024-06-18T02:28:19.083 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-06-18T02:28:19.083 INFO:teuthology.orchestra.run.smithi001.stdout:Total 3.0 MB/s | 2.2 MB 00:00 2024-06-18T02:28:19.119 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-06-18T02:28:19.120 INFO:teuthology.orchestra.run.smithi028.stdout:Nothing to do. 2024-06-18T02:28:19.120 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-06-18T02:28:19.147 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-06-18T02:28:19.163 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-06-18T02:28:19.163 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-06-18T02:28:19.177 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install libcephfs-devel 2024-06-18T02:28:19.255 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-06-18T02:28:19.255 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-06-18T02:28:19.778 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:34 ago on Tue 18 Jun 2024 02:26:45 AM UTC. 2024-06-18T02:28:19.889 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-06-18T02:28:19.911 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-06-18T02:28:19.912 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-06-18T02:28:19.912 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repo Size 2024-06-18T02:28:19.912 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-06-18T02:28:19.912 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-06-18T02:28:19.912 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs-devel x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 25 k 2024-06-18T02:28:19.912 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:28:19.912 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-06-18T02:28:19.912 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-06-18T02:28:19.912 INFO:teuthology.orchestra.run.smithi028.stdout:Install 1 Package 2024-06-18T02:28:19.912 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:28:19.912 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 25 k 2024-06-18T02:28:19.913 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 110 k 2024-06-18T02:28:19.913 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-06-18T02:28:20.053 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-06-18T02:28:20.108 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-06-18T02:28:20.162 INFO:teuthology.orchestra.run.smithi028.stdout:libcephfs-devel-17.2.7-1012.g3fe41055.el9.x86_6 100 kB/s | 25 kB 00:00 2024-06-18T02:28:20.162 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-06-18T02:28:20.162 INFO:teuthology.orchestra.run.smithi028.stdout:Total 100 kB/s | 25 kB 00:00 2024-06-18T02:28:20.162 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-06-18T02:28:20.166 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-06-18T02:28:20.166 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-06-18T02:28:20.168 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-06-18T02:28:20.168 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-06-18T02:28:20.173 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-06-18T02:28:20.208 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-06-18T02:28:20.250 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-06-18T02:28:20.252 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-06-18T02:28:20.351 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-06-18T02:28:20.361 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libcephfs-devel-2:17.2.7-1012.g3fe41055.el9.x86_64 1/1 2024-06-18T02:28:20.458 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-google-auth-1:2.30.0-1.el9.noarch 7/13 2024-06-18T02:28:20.519 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-06-18T02:28:20.702 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1012.g3fe41055.el9.x86_64 1/1 2024-06-18T02:28:20.922 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libcephfs-devel-2:17.2.7-1012.g3fe41055.el9.x86_64 1/1 2024-06-18T02:28:20.922 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:28:20.922 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-06-18T02:28:20.922 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs-devel-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:28:20.922 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:28:20.922 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-06-18T02:28:21.102 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install librados2 2024-06-18T02:28:21.278 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-06-18T02:28:21.331 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-06-18T02:28:21.365 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-06-18T02:28:21.411 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-06-18T02:28:21.459 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-rook-2:17.2.7-1012.g3fe41055.el9.noarch 13/13 2024-06-18T02:28:21.699 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:36 ago on Tue 18 Jun 2024 02:26:45 AM UTC. 2024-06-18T02:28:21.787 INFO:teuthology.orchestra.run.smithi028.stdout:Package librados2-2:17.2.7-1012.g3fe41055.el9.x86_64 is already installed. 2024-06-18T02:28:21.831 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-06-18T02:28:21.833 INFO:teuthology.orchestra.run.smithi028.stdout:Nothing to do. 2024-06-18T02:28:21.833 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-06-18T02:28:21.895 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install librbd1 2024-06-18T02:28:22.229 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1012.g3fe41055.el9.noarch 13/13 2024-06-18T02:28:22.229 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1012.g3fe41055.el9.noarch 1/13 2024-06-18T02:28:22.229 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-06-18T02:28:22.229 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-06-18T02:28:22.229 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-06-18T02:28:22.229 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-06-18T02:28:22.229 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-06-18T02:28:22.230 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-06-18T02:28:22.230 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-06-18T02:28:22.230 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-06-18T02:28:22.230 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-google-auth-1:2.30.0-1.el9.noarch 10/13 2024-06-18T02:28:22.230 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-06-18T02:28:22.230 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-06-18T02:28:22.491 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:37 ago on Tue 18 Jun 2024 02:26:45 AM UTC. 2024-06-18T02:28:22.579 INFO:teuthology.orchestra.run.smithi028.stdout:Package librbd1-2:17.2.7-1012.g3fe41055.el9.x86_64 is already installed. 2024-06-18T02:28:22.623 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-06-18T02:28:22.624 INFO:teuthology.orchestra.run.smithi028.stdout:Nothing to do. 2024-06-18T02:28:22.624 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-06-18T02:28:22.687 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-06-18T02:28:22.687 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:28:22.688 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-06-18T02:28:22.688 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-rook-2:17.2.7-1012.g3fe41055.el9.noarch 2024-06-18T02:28:22.688 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-06-18T02:28:22.688 INFO:teuthology.orchestra.run.smithi001.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-06-18T02:28:22.688 INFO:teuthology.orchestra.run.smithi001.stdout: python3-google-auth-1:2.30.0-1.el9.noarch 2024-06-18T02:28:22.688 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-06-18T02:28:22.688 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-06-18T02:28:22.688 INFO:teuthology.orchestra.run.smithi001.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-06-18T02:28:22.688 INFO:teuthology.orchestra.run.smithi001.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-06-18T02:28:22.688 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-06-18T02:28:22.688 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-06-18T02:28:22.688 INFO:teuthology.orchestra.run.smithi001.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-06-18T02:28:22.688 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rsa-4.9-2.el9.noarch 2024-06-18T02:28:22.689 INFO:teuthology.orchestra.run.smithi001.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-06-18T02:28:22.689 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:28:22.689 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-06-18T02:28:22.691 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install python3-rados 2024-06-18T02:28:22.928 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-mgr-cephadm 2024-06-18T02:28:23.291 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:38 ago on Tue 18 Jun 2024 02:26:45 AM UTC. 2024-06-18T02:28:23.378 INFO:teuthology.orchestra.run.smithi028.stdout:Package python3-rados-2:17.2.7-1012.g3fe41055.el9.x86_64 is already installed. 2024-06-18T02:28:23.422 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-06-18T02:28:23.423 INFO:teuthology.orchestra.run.smithi028.stdout:Nothing to do. 2024-06-18T02:28:23.423 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-06-18T02:28:23.483 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install python3-rgw 2024-06-18T02:28:23.524 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:22 ago on Tue 18 Jun 2024 02:27:01 AM UTC. 2024-06-18T02:28:23.655 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-06-18T02:28:23.656 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-06-18T02:28:23.656 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-06-18T02:28:23.656 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-06-18T02:28:23.656 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-06-18T02:28:23.656 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1012.g3fe41055.el9 ceph-noarch 123 k 2024-06-18T02:28:23.657 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-06-18T02:28:23.657 INFO:teuthology.orchestra.run.smithi001.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-06-18T02:28:23.657 INFO:teuthology.orchestra.run.smithi001.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-06-18T02:28:23.657 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-06-18T02:28:23.657 INFO:teuthology.orchestra.run.smithi001.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-06-18T02:28:23.657 INFO:teuthology.orchestra.run.smithi001.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-06-18T02:28:23.657 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:28:23.657 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-06-18T02:28:23.657 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-06-18T02:28:23.657 INFO:teuthology.orchestra.run.smithi001.stdout:Install 6 Packages 2024-06-18T02:28:23.657 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:28:23.658 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 7.0 M 2024-06-18T02:28:23.658 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 33 M 2024-06-18T02:28:23.658 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-06-18T02:28:24.080 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:39 ago on Tue 18 Jun 2024 02:26:45 AM UTC. 2024-06-18T02:28:24.168 INFO:teuthology.orchestra.run.smithi028.stdout:Package python3-rgw-2:17.2.7-1012.g3fe41055.el9.x86_64 is already installed. 2024-06-18T02:28:24.217 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-06-18T02:28:24.218 INFO:teuthology.orchestra.run.smithi028.stdout:Nothing to do. 2024-06-18T02:28:24.218 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-06-18T02:28:24.278 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install python3-cephfs 2024-06-18T02:28:24.307 INFO:teuthology.orchestra.run.smithi001.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.7 MB/s | 248 kB 00:00 2024-06-18T02:28:24.374 INFO:teuthology.orchestra.run.smithi001.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 8.0 MB/s | 548 kB 00:00 2024-06-18T02:28:24.399 INFO:teuthology.orchestra.run.smithi001.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.3 MB/s | 58 kB 00:00 2024-06-18T02:28:24.424 INFO:teuthology.orchestra.run.smithi001.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 2.0 MB/s | 51 kB 00:00 2024-06-18T02:28:24.517 INFO:teuthology.orchestra.run.smithi001.stdout:(5/6): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 6.0 MB 00:00 2024-06-18T02:28:24.542 INFO:teuthology.orchestra.run.smithi001.stdout:(6/6): ceph-mgr-cephadm-17.2.7-1012.g3fe41055.e 327 kB/s | 123 kB 00:00 2024-06-18T02:28:24.543 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-06-18T02:28:24.543 INFO:teuthology.orchestra.run.smithi001.stdout:Total 7.9 MB/s | 7.0 MB 00:00 2024-06-18T02:28:24.617 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-06-18T02:28:24.629 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-06-18T02:28:24.629 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-06-18T02:28:24.733 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-06-18T02:28:24.733 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-06-18T02:28:24.876 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:39 ago on Tue 18 Jun 2024 02:26:45 AM UTC. 2024-06-18T02:28:24.965 INFO:teuthology.orchestra.run.smithi028.stdout:Package python3-cephfs-2:17.2.7-1012.g3fe41055.el9.x86_64 is already installed. 2024-06-18T02:28:25.011 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-06-18T02:28:25.012 INFO:teuthology.orchestra.run.smithi028.stdout:Nothing to do. 2024-06-18T02:28:25.012 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-06-18T02:28:25.047 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-06-18T02:28:25.078 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install python3-rbd 2024-06-18T02:28:25.222 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-06-18T02:28:25.286 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-06-18T02:28:25.678 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:40 ago on Tue 18 Jun 2024 02:26:45 AM UTC. 2024-06-18T02:28:25.766 INFO:teuthology.orchestra.run.smithi028.stdout:Package python3-rbd-2:17.2.7-1012.g3fe41055.el9.x86_64 is already installed. 2024-06-18T02:28:25.810 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-06-18T02:28:25.811 INFO:teuthology.orchestra.run.smithi028.stdout:Nothing to do. 2024-06-18T02:28:25.811 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-06-18T02:28:25.870 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-06-18T02:28:25.876 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install rbd-fuse 2024-06-18T02:28:25.966 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-06-18T02:28:26.028 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-06-18T02:28:26.056 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1012.g3fe41055.el9.noarch 6/6 2024-06-18T02:28:26.473 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:41 ago on Tue 18 Jun 2024 02:26:45 AM UTC. 2024-06-18T02:28:26.609 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-06-18T02:28:26.609 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-06-18T02:28:26.610 INFO:teuthology.orchestra.run.smithi028.stdout: Package Architecture Version Repository Size 2024-06-18T02:28:26.610 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-06-18T02:28:26.610 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-06-18T02:28:26.610 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-fuse x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 82 k 2024-06-18T02:28:26.610 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:28:26.610 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-06-18T02:28:26.610 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-06-18T02:28:26.610 INFO:teuthology.orchestra.run.smithi028.stdout:Install 1 Package 2024-06-18T02:28:26.610 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:28:26.611 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 82 k 2024-06-18T02:28:26.611 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 222 k 2024-06-18T02:28:26.611 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-06-18T02:28:26.814 INFO:teuthology.orchestra.run.smithi028.stdout:rbd-fuse-17.2.7-1012.g3fe41055.el9.x86_64.rpm 402 kB/s | 82 kB 00:00 2024-06-18T02:28:26.816 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-06-18T02:28:26.816 INFO:teuthology.orchestra.run.smithi028.stdout:Total 399 kB/s | 82 kB 00:00 2024-06-18T02:28:26.816 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-06-18T02:28:26.816 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1012.g3fe41055.el9.noarch 6/6 2024-06-18T02:28:26.816 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1012.g3fe41055.el9.noarch 1/6 2024-06-18T02:28:26.816 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-06-18T02:28:26.816 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-06-18T02:28:26.816 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-06-18T02:28:26.816 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-06-18T02:28:26.823 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-06-18T02:28:26.823 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-06-18T02:28:26.890 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-06-18T02:28:26.890 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-06-18T02:28:27.043 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-06-18T02:28:27.093 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-06-18T02:28:27.094 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:28:27.094 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-06-18T02:28:27.094 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-cephadm-2:17.2.7-1012.g3fe41055.el9.noarch 2024-06-18T02:28:27.094 INFO:teuthology.orchestra.run.smithi001.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-06-18T02:28:27.094 INFO:teuthology.orchestra.run.smithi001.stdout: python3-babel-2.9.1-2.el9.noarch 2024-06-18T02:28:27.094 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-06-18T02:28:27.094 INFO:teuthology.orchestra.run.smithi001.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-06-18T02:28:27.094 INFO:teuthology.orchestra.run.smithi001.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-06-18T02:28:27.094 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:28:27.094 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-06-18T02:28:27.139 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : rbd-fuse-2:17.2.7-1012.g3fe41055.el9.x86_64 1/1 2024-06-18T02:28:27.340 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-fuse 2024-06-18T02:28:27.489 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1012.g3fe41055.el9.x86_64 1/1 2024-06-18T02:28:27.731 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : rbd-fuse-2:17.2.7-1012.g3fe41055.el9.x86_64 1/1 2024-06-18T02:28:27.731 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:28:27.731 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-06-18T02:28:27.732 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-fuse-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:28:27.732 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:28:27.732 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-06-18T02:28:27.950 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install rbd-mirror 2024-06-18T02:28:27.987 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:26 ago on Tue 18 Jun 2024 02:27:01 AM UTC. 2024-06-18T02:28:28.118 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-06-18T02:28:28.119 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-06-18T02:28:28.119 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-06-18T02:28:28.119 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-06-18T02:28:28.119 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-06-18T02:28:28.119 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-fuse x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 812 k 2024-06-18T02:28:28.119 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-06-18T02:28:28.119 INFO:teuthology.orchestra.run.smithi001.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-06-18T02:28:28.119 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:28:28.119 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-06-18T02:28:28.119 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-06-18T02:28:28.120 INFO:teuthology.orchestra.run.smithi001.stdout:Install 2 Packages 2024-06-18T02:28:28.120 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:28:28.120 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 892 k 2024-06-18T02:28:28.120 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 2.6 M 2024-06-18T02:28:28.120 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-06-18T02:28:28.360 INFO:teuthology.orchestra.run.smithi001.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 463 kB/s | 80 kB 00:00 2024-06-18T02:28:28.553 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:43 ago on Tue 18 Jun 2024 02:26:45 AM UTC. 2024-06-18T02:28:28.577 INFO:teuthology.orchestra.run.smithi001.stdout:(2/2): ceph-fuse-17.2.7-1012.g3fe41055.el9.x86_ 2.0 MB/s | 812 kB 00:00 2024-06-18T02:28:28.577 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-06-18T02:28:28.578 INFO:teuthology.orchestra.run.smithi001.stdout:Total 1.9 MB/s | 892 kB 00:00 2024-06-18T02:28:28.590 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-06-18T02:28:28.613 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-06-18T02:28:28.613 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-06-18T02:28:28.688 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-06-18T02:28:28.689 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-06-18T02:28:28.689 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repo Size 2024-06-18T02:28:28.689 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-06-18T02:28:28.689 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-06-18T02:28:28.689 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-mirror x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 2.9 M 2024-06-18T02:28:28.689 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:28:28.689 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-06-18T02:28:28.689 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-06-18T02:28:28.689 INFO:teuthology.orchestra.run.smithi028.stdout:Install 1 Package 2024-06-18T02:28:28.689 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:28:28.690 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 2.9 M 2024-06-18T02:28:28.690 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 12 M 2024-06-18T02:28:28.690 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-06-18T02:28:28.696 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-06-18T02:28:28.696 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-06-18T02:28:28.887 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-06-18T02:28:29.035 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-06-18T02:28:29.104 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-fuse-2:17.2.7-1012.g3fe41055.el9.x86_64 2/2 2024-06-18T02:28:29.350 INFO:teuthology.orchestra.run.smithi028.stdout:rbd-mirror-17.2.7-1012.g3fe41055.el9.x86_64.rpm 4.4 MB/s | 2.9 MB 00:00 2024-06-18T02:28:29.351 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-06-18T02:28:29.351 INFO:teuthology.orchestra.run.smithi028.stdout:Total 4.4 MB/s | 2.9 MB 00:00 2024-06-18T02:28:29.351 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-06-18T02:28:29.362 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-06-18T02:28:29.362 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-06-18T02:28:29.446 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-06-18T02:28:29.446 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-06-18T02:28:29.750 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1012.g3fe41055.el9.x86_64 2/2 2024-06-18T02:28:29.750 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-fuse-2:17.2.7-1012.g3fe41055.el9.x86_64 1/2 2024-06-18T02:28:29.886 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-06-18T02:28:29.968 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : rbd-mirror-2:17.2.7-1012.g3fe41055.el9.x86_64 1/1 2024-06-18T02:28:29.994 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1012.g3fe41055.el9.x86_64 1/1 2024-06-18T02:28:29.994 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-18T02:28:29.994 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-06-18T02:28:29.994 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-18T02:28:29.994 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-18T02:28:29.994 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:28:30.053 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-06-18T02:28:30.053 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:28:30.053 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-06-18T02:28:30.053 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-fuse-2:17.2.7-1012.g3fe41055.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-06-18T02:28:30.053 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:28:30.053 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-06-18T02:28:30.323 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install librados-devel 2024-06-18T02:28:30.923 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:29 ago on Tue 18 Jun 2024 02:27:01 AM UTC. 2024-06-18T02:28:31.055 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : rbd-mirror-2:17.2.7-1012.g3fe41055.el9.x86_64 1/1 2024-06-18T02:28:31.055 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:28:31.055 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-06-18T02:28:31.055 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-mirror-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:28:31.055 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:28:31.055 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-06-18T02:28:31.057 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-06-18T02:28:31.057 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-06-18T02:28:31.057 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repo Size 2024-06-18T02:28:31.057 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-06-18T02:28:31.058 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-06-18T02:28:31.058 INFO:teuthology.orchestra.run.smithi001.stdout: librados-devel x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 126 k 2024-06-18T02:28:31.058 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:28:31.058 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-06-18T02:28:31.058 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-06-18T02:28:31.058 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-06-18T02:28:31.058 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:28:31.059 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 126 k 2024-06-18T02:28:31.062 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 448 k 2024-06-18T02:28:31.062 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-06-18T02:28:31.255 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install rbd-nbd 2024-06-18T02:28:31.290 INFO:teuthology.orchestra.run.smithi001.stdout:librados-devel-17.2.7-1012.g3fe41055.el9.x86_64 547 kB/s | 126 kB 00:00 2024-06-18T02:28:31.291 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-06-18T02:28:31.291 INFO:teuthology.orchestra.run.smithi001.stdout:Total 543 kB/s | 126 kB 00:00 2024-06-18T02:28:31.291 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-06-18T02:28:31.297 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-06-18T02:28:31.297 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-06-18T02:28:31.363 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-06-18T02:28:31.363 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-06-18T02:28:31.520 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-06-18T02:28:31.633 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : librados-devel-2:17.2.7-1012.g3fe41055.el9.x86_64 1/1 2024-06-18T02:28:31.859 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:46 ago on Tue 18 Jun 2024 02:26:45 AM UTC. 2024-06-18T02:28:31.992 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-06-18T02:28:31.993 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-06-18T02:28:31.993 INFO:teuthology.orchestra.run.smithi028.stdout: Package Architecture Version Repository Size 2024-06-18T02:28:31.993 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-06-18T02:28:31.993 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-06-18T02:28:31.993 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-nbd x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 165 k 2024-06-18T02:28:31.993 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:28:31.993 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-06-18T02:28:31.993 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-06-18T02:28:31.993 INFO:teuthology.orchestra.run.smithi028.stdout:Install 1 Package 2024-06-18T02:28:31.993 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:28:31.993 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 165 k 2024-06-18T02:28:31.993 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 477 k 2024-06-18T02:28:31.994 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-06-18T02:28:32.023 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librados-devel-2:17.2.7-1012.g3fe41055.el9.x86_64 1/1 2024-06-18T02:28:32.303 INFO:teuthology.orchestra.run.smithi028.stdout:rbd-nbd-17.2.7-1012.g3fe41055.el9.x86_64.rpm 533 kB/s | 165 kB 00:00 2024-06-18T02:28:32.304 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-06-18T02:28:32.304 INFO:teuthology.orchestra.run.smithi028.stdout:Total 531 kB/s | 165 kB 00:00 2024-06-18T02:28:32.304 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-06-18T02:28:32.309 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librados-devel-2:17.2.7-1012.g3fe41055.el9.x86_64 1/1 2024-06-18T02:28:32.309 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:28:32.309 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-06-18T02:28:32.310 INFO:teuthology.orchestra.run.smithi001.stdout: librados-devel-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:28:32.310 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:28:32.310 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-06-18T02:28:32.313 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-06-18T02:28:32.313 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-06-18T02:28:32.379 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-06-18T02:28:32.379 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-06-18T02:28:32.536 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-06-18T02:28:32.559 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install libcephfs2 2024-06-18T02:28:32.636 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : rbd-nbd-2:17.2.7-1012.g3fe41055.el9.x86_64 1/1 2024-06-18T02:28:33.027 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1012.g3fe41055.el9.x86_64 1/1 2024-06-18T02:28:33.160 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:32 ago on Tue 18 Jun 2024 02:27:01 AM UTC. 2024-06-18T02:28:33.251 INFO:teuthology.orchestra.run.smithi001.stdout:Package libcephfs2-2:17.2.7-1012.g3fe41055.el9.x86_64 is already installed. 2024-06-18T02:28:33.289 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : rbd-nbd-2:17.2.7-1012.g3fe41055.el9.x86_64 1/1 2024-06-18T02:28:33.289 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:28:33.289 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-06-18T02:28:33.290 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-nbd-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:28:33.290 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:28:33.290 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-06-18T02:28:33.296 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-06-18T02:28:33.297 INFO:teuthology.orchestra.run.smithi001.stdout:Nothing to do. 2024-06-18T02:28:33.297 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-06-18T02:28:33.372 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install libcephfs-devel 2024-06-18T02:28:33.532 DEBUG:teuthology.parallel:result is None 2024-06-18T02:28:33.970 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:32 ago on Tue 18 Jun 2024 02:27:01 AM UTC. 2024-06-18T02:28:34.102 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-06-18T02:28:34.103 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-06-18T02:28:34.103 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repo Size 2024-06-18T02:28:34.103 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-06-18T02:28:34.103 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-06-18T02:28:34.103 INFO:teuthology.orchestra.run.smithi001.stdout: libcephfs-devel x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 25 k 2024-06-18T02:28:34.103 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:28:34.103 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-06-18T02:28:34.103 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-06-18T02:28:34.104 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-06-18T02:28:34.104 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:28:34.104 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 25 k 2024-06-18T02:28:34.104 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 110 k 2024-06-18T02:28:34.104 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-06-18T02:28:34.344 INFO:teuthology.orchestra.run.smithi001.stdout:libcephfs-devel-17.2.7-1012.g3fe41055.el9.x86_6 104 kB/s | 25 kB 00:00 2024-06-18T02:28:34.344 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-06-18T02:28:34.344 INFO:teuthology.orchestra.run.smithi001.stdout:Total 103 kB/s | 25 kB 00:00 2024-06-18T02:28:34.345 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-06-18T02:28:34.348 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-06-18T02:28:34.349 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-06-18T02:28:34.350 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-06-18T02:28:34.350 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-06-18T02:28:34.430 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-06-18T02:28:34.536 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libcephfs-devel-2:17.2.7-1012.g3fe41055.el9.x86_64 1/1 2024-06-18T02:28:34.931 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1012.g3fe41055.el9.x86_64 1/1 2024-06-18T02:28:35.167 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libcephfs-devel-2:17.2.7-1012.g3fe41055.el9.x86_64 1/1 2024-06-18T02:28:35.167 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:28:35.168 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-06-18T02:28:35.168 INFO:teuthology.orchestra.run.smithi001.stdout: libcephfs-devel-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:28:35.168 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:28:35.168 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-06-18T02:28:35.361 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install librados2 2024-06-18T02:28:35.957 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:34 ago on Tue 18 Jun 2024 02:27:01 AM UTC. 2024-06-18T02:28:36.047 INFO:teuthology.orchestra.run.smithi001.stdout:Package librados2-2:17.2.7-1012.g3fe41055.el9.x86_64 is already installed. 2024-06-18T02:28:36.091 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-06-18T02:28:36.092 INFO:teuthology.orchestra.run.smithi001.stdout:Nothing to do. 2024-06-18T02:28:36.092 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-06-18T02:28:36.173 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install librbd1 2024-06-18T02:28:36.765 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:35 ago on Tue 18 Jun 2024 02:27:01 AM UTC. 2024-06-18T02:28:36.852 INFO:teuthology.orchestra.run.smithi001.stdout:Package librbd1-2:17.2.7-1012.g3fe41055.el9.x86_64 is already installed. 2024-06-18T02:28:36.896 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-06-18T02:28:36.897 INFO:teuthology.orchestra.run.smithi001.stdout:Nothing to do. 2024-06-18T02:28:36.897 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-06-18T02:28:36.978 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install python3-rados 2024-06-18T02:28:37.574 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:36 ago on Tue 18 Jun 2024 02:27:01 AM UTC. 2024-06-18T02:28:37.661 INFO:teuthology.orchestra.run.smithi001.stdout:Package python3-rados-2:17.2.7-1012.g3fe41055.el9.x86_64 is already installed. 2024-06-18T02:28:37.705 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-06-18T02:28:37.705 INFO:teuthology.orchestra.run.smithi001.stdout:Nothing to do. 2024-06-18T02:28:37.706 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-06-18T02:28:37.779 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install python3-rgw 2024-06-18T02:28:38.376 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:37 ago on Tue 18 Jun 2024 02:27:01 AM UTC. 2024-06-18T02:28:38.464 INFO:teuthology.orchestra.run.smithi001.stdout:Package python3-rgw-2:17.2.7-1012.g3fe41055.el9.x86_64 is already installed. 2024-06-18T02:28:38.508 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-06-18T02:28:38.509 INFO:teuthology.orchestra.run.smithi001.stdout:Nothing to do. 2024-06-18T02:28:38.510 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-06-18T02:28:38.578 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install python3-cephfs 2024-06-18T02:28:39.174 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:38 ago on Tue 18 Jun 2024 02:27:01 AM UTC. 2024-06-18T02:28:39.262 INFO:teuthology.orchestra.run.smithi001.stdout:Package python3-cephfs-2:17.2.7-1012.g3fe41055.el9.x86_64 is already installed. 2024-06-18T02:28:39.305 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-06-18T02:28:39.306 INFO:teuthology.orchestra.run.smithi001.stdout:Nothing to do. 2024-06-18T02:28:39.306 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-06-18T02:28:39.386 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install python3-rbd 2024-06-18T02:28:39.985 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:38 ago on Tue 18 Jun 2024 02:27:01 AM UTC. 2024-06-18T02:28:40.072 INFO:teuthology.orchestra.run.smithi001.stdout:Package python3-rbd-2:17.2.7-1012.g3fe41055.el9.x86_64 is already installed. 2024-06-18T02:28:40.116 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-06-18T02:28:40.117 INFO:teuthology.orchestra.run.smithi001.stdout:Nothing to do. 2024-06-18T02:28:40.117 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-06-18T02:28:40.198 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install rbd-fuse 2024-06-18T02:28:40.796 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:39 ago on Tue 18 Jun 2024 02:27:01 AM UTC. 2024-06-18T02:28:40.929 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-06-18T02:28:40.929 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-06-18T02:28:40.929 INFO:teuthology.orchestra.run.smithi001.stdout: Package Architecture Version Repository Size 2024-06-18T02:28:40.929 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-06-18T02:28:40.929 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-06-18T02:28:40.929 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-fuse x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 82 k 2024-06-18T02:28:40.929 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:28:40.930 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-06-18T02:28:40.930 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-06-18T02:28:40.930 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-06-18T02:28:40.930 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:28:40.930 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 82 k 2024-06-18T02:28:40.930 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 222 k 2024-06-18T02:28:40.931 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-06-18T02:28:41.147 INFO:teuthology.orchestra.run.smithi001.stdout:rbd-fuse-17.2.7-1012.g3fe41055.el9.x86_64.rpm 378 kB/s | 82 kB 00:00 2024-06-18T02:28:41.147 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-06-18T02:28:41.148 INFO:teuthology.orchestra.run.smithi001.stdout:Total 376 kB/s | 82 kB 00:00 2024-06-18T02:28:41.148 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-06-18T02:28:41.155 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-06-18T02:28:41.155 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-06-18T02:28:41.220 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-06-18T02:28:41.221 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-06-18T02:28:41.371 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-06-18T02:28:41.480 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : rbd-fuse-2:17.2.7-1012.g3fe41055.el9.x86_64 1/1 2024-06-18T02:28:41.859 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1012.g3fe41055.el9.x86_64 1/1 2024-06-18T02:28:42.125 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : rbd-fuse-2:17.2.7-1012.g3fe41055.el9.x86_64 1/1 2024-06-18T02:28:42.125 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:28:42.125 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-06-18T02:28:42.125 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-fuse-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:28:42.125 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:28:42.125 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-06-18T02:28:42.366 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install rbd-mirror 2024-06-18T02:28:42.970 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:41 ago on Tue 18 Jun 2024 02:27:01 AM UTC. 2024-06-18T02:28:43.104 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-06-18T02:28:43.105 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-06-18T02:28:43.105 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repo Size 2024-06-18T02:28:43.105 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-06-18T02:28:43.105 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-06-18T02:28:43.105 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-mirror x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 2.9 M 2024-06-18T02:28:43.105 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:28:43.105 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-06-18T02:28:43.105 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-06-18T02:28:43.105 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-06-18T02:28:43.105 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:28:43.106 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 2.9 M 2024-06-18T02:28:43.106 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 12 M 2024-06-18T02:28:43.106 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-06-18T02:28:43.683 INFO:teuthology.orchestra.run.smithi001.stdout:rbd-mirror-17.2.7-1012.g3fe41055.el9.x86_64.rpm 5.1 MB/s | 2.9 MB 00:00 2024-06-18T02:28:43.684 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-06-18T02:28:43.684 INFO:teuthology.orchestra.run.smithi001.stdout:Total 5.1 MB/s | 2.9 MB 00:00 2024-06-18T02:28:43.684 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-06-18T02:28:43.695 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-06-18T02:28:43.695 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-06-18T02:28:43.779 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-06-18T02:28:43.780 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-06-18T02:28:44.214 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-06-18T02:28:44.284 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : rbd-mirror-2:17.2.7-1012.g3fe41055.el9.x86_64 1/1 2024-06-18T02:28:44.310 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1012.g3fe41055.el9.x86_64 1/1 2024-06-18T02:28:44.310 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-18T02:28:44.311 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-06-18T02:28:44.311 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-18T02:28:44.311 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-18T02:28:44.311 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:28:45.433 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : rbd-mirror-2:17.2.7-1012.g3fe41055.el9.x86_64 1/1 2024-06-18T02:28:45.433 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:28:45.433 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-06-18T02:28:45.433 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-mirror-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:28:45.433 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:28:45.433 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-06-18T02:28:45.678 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install rbd-nbd 2024-06-18T02:28:46.279 INFO:teuthology.orchestra.run.smithi001.stdout:Last metadata expiration check: 0:01:45 ago on Tue 18 Jun 2024 02:27:01 AM UTC. 2024-06-18T02:28:46.414 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-06-18T02:28:46.415 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-06-18T02:28:46.415 INFO:teuthology.orchestra.run.smithi001.stdout: Package Architecture Version Repository Size 2024-06-18T02:28:46.415 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-06-18T02:28:46.415 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-06-18T02:28:46.415 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-nbd x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 165 k 2024-06-18T02:28:46.415 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:28:46.415 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-06-18T02:28:46.415 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-06-18T02:28:46.416 INFO:teuthology.orchestra.run.smithi001.stdout:Install 1 Package 2024-06-18T02:28:46.416 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:28:46.416 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 165 k 2024-06-18T02:28:46.416 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 477 k 2024-06-18T02:28:46.416 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-06-18T02:28:46.771 INFO:teuthology.orchestra.run.smithi001.stdout:rbd-nbd-17.2.7-1012.g3fe41055.el9.x86_64.rpm 466 kB/s | 165 kB 00:00 2024-06-18T02:28:46.772 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-06-18T02:28:46.772 INFO:teuthology.orchestra.run.smithi001.stdout:Total 463 kB/s | 165 kB 00:00 2024-06-18T02:28:46.772 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-06-18T02:28:46.787 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-06-18T02:28:46.787 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-06-18T02:28:46.857 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-06-18T02:28:46.858 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-06-18T02:28:47.014 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-06-18T02:28:47.112 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : rbd-nbd-2:17.2.7-1012.g3fe41055.el9.x86_64 1/1 2024-06-18T02:28:47.524 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1012.g3fe41055.el9.x86_64 1/1 2024-06-18T02:28:47.793 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : rbd-nbd-2:17.2.7-1012.g3fe41055.el9.x86_64 1/1 2024-06-18T02:28:47.793 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:28:47.793 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-06-18T02:28:47.793 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-nbd-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:28:47.793 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:28:47.793 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-06-18T02:28:48.093 DEBUG:teuthology.parallel:result is None 2024-06-18T02:28:48.093 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-18T02:28:48.093 INFO:teuthology.packaging:ref: None 2024-06-18T02:28:48.093 INFO:teuthology.packaging:tag: None 2024-06-18T02:28:48.094 INFO:teuthology.packaging:branch: quincy 2024-06-18T02:28:48.094 INFO:teuthology.packaging:sha1: 3781e3a592387d16e8020f5db99cdee4198f6d00 2024-06-18T02:28:48.094 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-06-18T02:28:48.385 DEBUG:teuthology.orchestra.run.smithi001:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-06-18T02:28:48.408 INFO:teuthology.orchestra.run.smithi001.stdout:17.2.7-1012.g3fe41055.el9 2024-06-18T02:28:48.409 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1012.g3fe41055.el9 2024-06-18T02:28:48.409 INFO:teuthology.task.install:The correct ceph version 17.2.7-1012.g3fe41055 is installed. 2024-06-18T02:28:48.410 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-18T02:28:48.410 INFO:teuthology.packaging:ref: None 2024-06-18T02:28:48.411 INFO:teuthology.packaging:tag: None 2024-06-18T02:28:48.411 INFO:teuthology.packaging:branch: quincy 2024-06-18T02:28:48.411 INFO:teuthology.packaging:sha1: 3781e3a592387d16e8020f5db99cdee4198f6d00 2024-06-18T02:28:48.411 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-06-18T02:28:48.604 DEBUG:teuthology.orchestra.run.smithi028:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-06-18T02:28:48.631 INFO:teuthology.orchestra.run.smithi028.stdout:17.2.7-1012.g3fe41055.el9 2024-06-18T02:28:48.631 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1012.g3fe41055.el9 2024-06-18T02:28:48.631 INFO:teuthology.task.install:The correct ceph version 17.2.7-1012.g3fe41055 is installed. 2024-06-18T02:28:48.633 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-06-18T02:28:48.633 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-06-18T02:28:48.633 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-06-18T02:28:48.662 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-06-18T02:28:48.662 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-06-18T02:28:48.702 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-06-18T02:28:48.703 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-06-18T02:28:48.703 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/usr/bin/daemon-helper 2024-06-18T02:28:48.729 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-06-18T02:28:48.796 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-06-18T02:28:48.797 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/daemon-helper 2024-06-18T02:28:48.827 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-06-18T02:28:48.897 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-06-18T02:28:48.897 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-06-18T02:28:48.897 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/usr/bin/adjust-ulimits 2024-06-18T02:28:48.925 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-06-18T02:28:48.991 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-06-18T02:28:48.991 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/adjust-ulimits 2024-06-18T02:28:49.020 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-06-18T02:28:49.089 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-06-18T02:28:49.089 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-06-18T02:28:49.089 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/usr/bin/stdin-killer 2024-06-18T02:28:49.115 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-06-18T02:28:49.180 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-06-18T02:28:49.181 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/stdin-killer 2024-06-18T02:28:49.209 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-06-18T02:28:49.277 INFO:teuthology.run_tasks:Running task print... 2024-06-18T02:28:49.287 INFO:teuthology.task.print:**** done install task... 2024-06-18T02:28:49.287 INFO:teuthology.run_tasks:Running task cephadm... 2024-06-18T02:28:49.380 INFO:tasks.cephadm:Config: {'cephadm_branch': 'quincy', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}}, 'image': 'quay.ceph.io/ceph-ci/ceph:quincy', 'roleless': True, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'OSD_DOWN', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy'], 'sha1': '3781e3a592387d16e8020f5db99cdee4198f6d00'} 2024-06-18T02:28:49.380 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:quincy 2024-06-18T02:28:49.381 INFO:tasks.cephadm:Cluster fsid is 7eefb63e-2d1a-11ef-bc9d-c7b262605968 2024-06-18T02:28:49.381 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-06-18T02:28:49.381 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-06-18T02:28:49.381 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi001': '172.21.15.1', 'mon.smithi028': '172.21.15.28'} 2024-06-18T02:28:49.381 INFO:tasks.cephadm:Normalizing hostnames... 2024-06-18T02:28:49.382 DEBUG:teuthology.orchestra.run.smithi001:> sudo hostname $(hostname -s) 2024-06-18T02:28:49.408 DEBUG:teuthology.orchestra.run.smithi028:> sudo hostname $(hostname -s) 2024-06-18T02:28:49.437 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref quincy)... 2024-06-18T02:28:49.438 DEBUG:teuthology.orchestra.run.smithi001:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-06-18T02:28:49.656 INFO:teuthology.orchestra.run.smithi001.stdout:-rw-r--r--. 1 ubuntu ubuntu 375153 Jun 18 02:28 /home/ubuntu/cephtest/cephadm 2024-06-18T02:28:49.657 DEBUG:teuthology.orchestra.run.smithi028:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-06-18T02:28:49.839 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r--. 1 ubuntu ubuntu 375153 Jun 18 02:28 /home/ubuntu/cephtest/cephadm 2024-06-18T02:28:49.840 DEBUG:teuthology.orchestra.run.smithi001:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-06-18T02:28:49.857 DEBUG:teuthology.orchestra.run.smithi028:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-06-18T02:28:49.885 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:quincy on all hosts... 2024-06-18T02:28:49.885 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-06-18T02:28:49.900 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-06-18T02:28:50.783 INFO:teuthology.orchestra.run.smithi028.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-06-18T02:28:50.795 INFO:teuthology.orchestra.run.smithi001.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-06-18T02:29:49.048 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-06-18T02:29:49.048 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_version": "ceph version 17.2.7-1012-g3fe41055 (3fe410552b224c0fbe57dfbac3e18568c0f8fef6) quincy (stable)", 2024-06-18T02:29:49.048 INFO:teuthology.orchestra.run.smithi001.stdout: "image_id": "1a271607156e1b6becab587233eefa647399ff75aa2bb76fdb2f0360ec5b7818", 2024-06-18T02:29:49.048 INFO:teuthology.orchestra.run.smithi001.stdout: "repo_digests": [ 2024-06-18T02:29:49.049 INFO:teuthology.orchestra.run.smithi001.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032" 2024-06-18T02:29:49.049 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-06-18T02:29:49.049 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-06-18T02:29:49.357 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-06-18T02:29:49.357 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_version": "ceph version 17.2.7-1012-g3fe41055 (3fe410552b224c0fbe57dfbac3e18568c0f8fef6) quincy (stable)", 2024-06-18T02:29:49.357 INFO:teuthology.orchestra.run.smithi028.stdout: "image_id": "1a271607156e1b6becab587233eefa647399ff75aa2bb76fdb2f0360ec5b7818", 2024-06-18T02:29:49.357 INFO:teuthology.orchestra.run.smithi028.stdout: "repo_digests": [ 2024-06-18T02:29:49.357 INFO:teuthology.orchestra.run.smithi028.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032" 2024-06-18T02:29:49.357 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-06-18T02:29:49.357 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-06-18T02:29:49.377 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /etc/ceph 2024-06-18T02:29:49.410 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /etc/ceph 2024-06-18T02:29:49.438 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 777 /etc/ceph 2024-06-18T02:29:49.478 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 777 /etc/ceph 2024-06-18T02:29:49.505 INFO:tasks.cephadm:Writing seed config... 2024-06-18T02:29:49.506 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-06-18T02:29:49.506 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-06-18T02:29:49.507 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-06-18T02:29:49.507 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-06-18T02:29:49.507 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-06-18T02:29:49.507 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-06-18T02:29:49.507 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-06-18T02:29:49.507 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-06-18T02:29:49.507 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-06-18T02:29:49.507 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-06-18T02:29:49.507 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-06-18T02:29:49.507 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-06-18T02:29:49.507 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-06-18T02:29:49.507 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-06-18T02:29:49.507 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-06-18T02:29:49.507 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-06-18T02:29:49.508 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-06-18T02:29:49.508 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-06-18T02:29:49.508 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-06-18T02:29:49.508 INFO:tasks.cephadm: override: [client] debug client = 20 2024-06-18T02:29:49.508 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-06-18T02:29:49.508 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-06-18T02:29:49.508 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-06-18T02:29:49.508 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-06-18T02:29:49.508 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-06-18T02:29:49.508 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-06-18T02:29:49.508 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-06-18T02:29:49.508 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-06-18T02:29:49.508 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-06-18T02:29:49.508 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-06-18T02:29:49.509 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-06-18T02:29:49.509 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-06-18T02:29:49.509 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-06-18T02:29:49.509 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-06-18T02:29:49.509 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-06-18T02:29:49.509 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-06-18T02:29:49.509 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-06-18T02:29:49.509 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-06-18T02:29:49.509 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-06-18T02:29:49.509 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-06-18T02:29:49.509 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-06-18T02:29:49.509 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-06-18T02:29:49.510 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-06-18T02:29:49.510 DEBUG:teuthology.orchestra.run.smithi001:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-06-18T02:29:49.535 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 7eefb63e-2d1a-11ef-bc9d-c7b262605968 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-06-18T02:29:49.535 DEBUG:teuthology.orchestra.run.smithi001:mon.smithi001> sudo journalctl -f -n 0 -u ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968@mon.smithi001.service 2024-06-18T02:29:49.577 INFO:tasks.cephadm:Bootstrapping... 2024-06-18T02:29:49.578 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy -v bootstrap --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.1 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-06-18T02:29:49.795 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-06-18T02:29:49.795 INFO:teuthology.orchestra.run.smithi001.stdout:cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:quincy', '-v', 'bootstrap', '--fsid', '7eefb63e-2d1a-11ef-bc9d-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.1', '--skip-admin-label'] 2024-06-18T02:29:49.820 INFO:teuthology.orchestra.run.smithi001.stdout:/bin/podman: stdout 5.1.0 2024-06-18T02:29:49.821 INFO:teuthology.orchestra.run.smithi001.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-06-18T02:29:49.821 INFO:teuthology.orchestra.run.smithi001.stdout:Verifying podman|docker is present... 2024-06-18T02:29:49.845 INFO:teuthology.orchestra.run.smithi001.stdout:/bin/podman: stdout 5.1.0 2024-06-18T02:29:49.845 INFO:teuthology.orchestra.run.smithi001.stdout:Verifying lvm2 is present... 2024-06-18T02:29:49.845 INFO:teuthology.orchestra.run.smithi001.stdout:Verifying time synchronization is in place... 2024-06-18T02:29:49.853 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-06-18T02:29:49.853 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-06-18T02:29:49.860 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-06-18T02:29:49.860 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stdout inactive 2024-06-18T02:29:49.868 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stdout enabled 2024-06-18T02:29:49.875 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stdout active 2024-06-18T02:29:49.875 INFO:teuthology.orchestra.run.smithi001.stdout:Unit chronyd.service is enabled and running 2024-06-18T02:29:49.875 INFO:teuthology.orchestra.run.smithi001.stdout:Repeating the final host check... 2024-06-18T02:29:49.899 INFO:teuthology.orchestra.run.smithi001.stdout:/bin/podman: stdout 5.1.0 2024-06-18T02:29:49.899 INFO:teuthology.orchestra.run.smithi001.stdout:podman (/bin/podman) version 5.1.0 is present 2024-06-18T02:29:49.899 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl is present 2024-06-18T02:29:49.899 INFO:teuthology.orchestra.run.smithi001.stdout:lvcreate is present 2024-06-18T02:29:49.906 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-06-18T02:29:49.906 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-06-18T02:29:49.913 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-06-18T02:29:49.913 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stdout inactive 2024-06-18T02:29:49.920 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stdout enabled 2024-06-18T02:29:49.927 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stdout active 2024-06-18T02:29:49.927 INFO:teuthology.orchestra.run.smithi001.stdout:Unit chronyd.service is enabled and running 2024-06-18T02:29:49.927 INFO:teuthology.orchestra.run.smithi001.stdout:Host looks OK 2024-06-18T02:29:49.927 INFO:teuthology.orchestra.run.smithi001.stdout:Cluster fsid: 7eefb63e-2d1a-11ef-bc9d-c7b262605968 2024-06-18T02:29:49.927 INFO:teuthology.orchestra.run.smithi001.stdout:Acquiring lock 139815499514304 on /run/cephadm/7eefb63e-2d1a-11ef-bc9d-c7b262605968.lock 2024-06-18T02:29:49.927 INFO:teuthology.orchestra.run.smithi001.stdout:Lock 139815499514304 acquired on /run/cephadm/7eefb63e-2d1a-11ef-bc9d-c7b262605968.lock 2024-06-18T02:29:49.928 INFO:teuthology.orchestra.run.smithi001.stdout:Verifying IP 172.21.15.1 port 3300 ... 2024-06-18T02:29:49.928 INFO:teuthology.orchestra.run.smithi001.stdout:Verifying IP 172.21.15.1 port 6789 ... 2024-06-18T02:29:49.928 INFO:teuthology.orchestra.run.smithi001.stdout:Base mon IP(s) is [172.21.15.1:3300, 172.21.15.1:6789], mon addrv is [v2:172.21.15.1:3300,v1:172.21.15.1:6789] 2024-06-18T02:29:49.932 INFO:teuthology.orchestra.run.smithi001.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.1 metric 100 2024-06-18T02:29:49.932 INFO:teuthology.orchestra.run.smithi001.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.1 metric 100 2024-06-18T02:29:49.935 INFO:teuthology.orchestra.run.smithi001.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-06-18T02:29:49.935 INFO:teuthology.orchestra.run.smithi001.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-06-18T02:29:49.935 INFO:teuthology.orchestra.run.smithi001.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-06-18T02:29:49.938 INFO:teuthology.orchestra.run.smithi001.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-06-18T02:29:49.938 INFO:teuthology.orchestra.run.smithi001.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-06-18T02:29:49.938 INFO:teuthology.orchestra.run.smithi001.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-06-18T02:29:49.938 INFO:teuthology.orchestra.run.smithi001.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-06-18T02:29:49.939 INFO:teuthology.orchestra.run.smithi001.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:15e8/64 scope link noprefixroute 2024-06-18T02:29:49.939 INFO:teuthology.orchestra.run.smithi001.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-06-18T02:29:49.940 INFO:teuthology.orchestra.run.smithi001.stdout:Mon IP `172.21.15.1` is in CIDR network `172.21.0.0/20` 2024-06-18T02:29:49.940 INFO:teuthology.orchestra.run.smithi001.stdout:Mon IP `172.21.15.1` is in CIDR network `172.21.0.0/20` 2024-06-18T02:29:49.940 INFO:teuthology.orchestra.run.smithi001.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-06-18T02:29:49.940 INFO:teuthology.orchestra.run.smithi001.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-06-18T02:29:49.940 INFO:teuthology.orchestra.run.smithi001.stdout:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-06-18T02:29:50.564 INFO:teuthology.orchestra.run.smithi001.stdout:/bin/podman: stdout 1a271607156e1b6becab587233eefa647399ff75aa2bb76fdb2f0360ec5b7818 2024-06-18T02:29:50.564 INFO:teuthology.orchestra.run.smithi001.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:quincy... 2024-06-18T02:29:50.564 INFO:teuthology.orchestra.run.smithi001.stdout:/bin/podman: stderr Getting image source signatures 2024-06-18T02:29:50.564 INFO:teuthology.orchestra.run.smithi001.stdout:/bin/podman: stderr Copying blob sha256:7409f993fad97651003c683d751391d26d5a62a9a56c7e456e77f6286b65bea9 2024-06-18T02:29:50.565 INFO:teuthology.orchestra.run.smithi001.stdout:/bin/podman: stderr Copying blob sha256:9065cd298533077f0006921bb12bd331d24f6c140cc8fa972c2e6c4e382038d1 2024-06-18T02:29:50.565 INFO:teuthology.orchestra.run.smithi001.stdout:/bin/podman: stderr Copying config sha256:1a271607156e1b6becab587233eefa647399ff75aa2bb76fdb2f0360ec5b7818 2024-06-18T02:29:50.565 INFO:teuthology.orchestra.run.smithi001.stdout:/bin/podman: stderr Writing manifest to image destination 2024-06-18T02:29:51.444 INFO:teuthology.orchestra.run.smithi001.stdout:ceph: stdout ceph version 17.2.7-1012-g3fe41055 (3fe410552b224c0fbe57dfbac3e18568c0f8fef6) quincy (stable) 2024-06-18T02:29:51.444 INFO:teuthology.orchestra.run.smithi001.stdout:Ceph version: ceph version 17.2.7-1012-g3fe41055 (3fe410552b224c0fbe57dfbac3e18568c0f8fef6) quincy (stable) 2024-06-18T02:29:51.444 INFO:teuthology.orchestra.run.smithi001.stdout:Extracting ceph user uid/gid from container image... 2024-06-18T02:29:52.393 INFO:teuthology.orchestra.run.smithi001.stdout:stat: stdout 167 167 2024-06-18T02:29:52.393 INFO:teuthology.orchestra.run.smithi001.stdout:Creating initial keys... 2024-06-18T02:29:53.273 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph-authtool: stdout AQAg8XBm8FinKBAAjVE2seHBoehdMgRhi1v+jw== 2024-06-18T02:29:54.136 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph-authtool: stdout AQAh8XBmlRSLJBAADCWJvL3WjTsZiqrtLVXwPg== 2024-06-18T02:29:55.125 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph-authtool: stdout AQAi8XBmqqx5HhAAsQ6Vtc068qT7+1oya2s6dA== 2024-06-18T02:29:55.125 INFO:teuthology.orchestra.run.smithi001.stdout:Creating initial monmap... 2024-06-18T02:29:56.065 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-06-18T02:29:56.065 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = octopus 2024-06-18T02:29:56.065 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 7eefb63e-2d1a-11ef-bc9d-c7b262605968 2024-06-18T02:29:56.066 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-06-18T02:29:56.066 INFO:teuthology.orchestra.run.smithi001.stdout:monmaptool for smithi001 [v2:172.21.15.1:3300,v1:172.21.15.1:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-06-18T02:29:56.066 INFO:teuthology.orchestra.run.smithi001.stdout:setting min_mon_release = octopus 2024-06-18T02:29:56.066 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/monmaptool: set fsid to 7eefb63e-2d1a-11ef-bc9d-c7b262605968 2024-06-18T02:29:56.066 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-06-18T02:29:56.066 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:29:56.066 INFO:teuthology.orchestra.run.smithi001.stdout:Creating mon... 2024-06-18T02:29:56.962 INFO:teuthology.orchestra.run.smithi001.stdout:create mon.smithi001 on 2024-06-18T02:29:57.178 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-06-18T02:29:57.343 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-06-18T02:29:57.567 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968.target → /etc/systemd/system/ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968.target. 2024-06-18T02:29:57.567 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968.target → /etc/systemd/system/ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968.target. 2024-06-18T02:29:57.769 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968@mon.smithi001 2024-06-18T02:29:57.769 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stderr Failed to reset failed state of unit ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968@mon.smithi001.service: Unit ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968@mon.smithi001.service not loaded. 2024-06-18T02:29:57.955 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968.target.wants/ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968@mon.smithi001.service → /etc/systemd/system/ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968@.service. 2024-06-18T02:29:58.007 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:29:57 smithi001 systemd[1]: Starting Ceph mon.smithi001 for 7eefb63e-2d1a-11ef-bc9d-c7b262605968... 2024-06-18T02:29:58.326 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:29:58 smithi001 podman[28705]: 2024-06-18 02:29:58.11126208 +0000 UTC m=+0.020323638 image pull 1a271607156e1b6becab587233eefa647399ff75aa2bb76fdb2f0360ec5b7818 quay.ceph.io/ceph-ci/ceph:quincy 2024-06-18T02:29:58.327 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:29:58 smithi001 podman[28705]: 2024-06-18 02:29:58.211934161 +0000 UTC m=+0.120995701 container create 74982d89df8650da93103cd9126a631e562c24a212905c96273187f58f5045c3 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-mon-smithi001, org.label-schema.build-date=20240612, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, RELEASE=quincy-3fe4105, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-06-18T02:29:58.546 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-06-18T02:29:58.546 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stdout disabled 2024-06-18T02:29:58.555 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-06-18T02:29:58.555 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stdout inactive 2024-06-18T02:29:58.555 INFO:teuthology.orchestra.run.smithi001.stdout:firewalld.service is not enabled 2024-06-18T02:29:58.555 INFO:teuthology.orchestra.run.smithi001.stdout:Not possible to enable service . firewalld.service is not available 2024-06-18T02:29:58.556 INFO:teuthology.orchestra.run.smithi001.stdout:Waiting for mon to start... 2024-06-18T02:29:58.556 INFO:teuthology.orchestra.run.smithi001.stdout:Waiting for mon... 2024-06-18T02:29:58.590 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:29:58 smithi001 podman[28705]: 2024-06-18 02:29:58.430735799 +0000 UTC m=+0.339797350 container init 74982d89df8650da93103cd9126a631e562c24a212905c96273187f58f5045c3 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-mon-smithi001, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-3fe4105, org.label-schema.build-date=20240612, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-18T02:29:58.590 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:29:58 smithi001 podman[28705]: 2024-06-18 02:29:58.434089843 +0000 UTC m=+0.343151383 container start 74982d89df8650da93103cd9126a631e562c24a212905c96273187f58f5045c3 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-mon-smithi001, io.buildah.version=1.36.0, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=quincy-3fe4105, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240612, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS) 2024-06-18T02:29:58.591 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:29:58 smithi001 ceph-mon[28724]: mkfs 7eefb63e-2d1a-11ef-bc9d-c7b262605968 2024-06-18T02:29:58.591 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:29:58 smithi001 ceph-mon[28724]: mon.smithi001 is new leader, mons smithi001 in quorum (ranks 0) 2024-06-18T02:29:58.591 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:29:58 smithi001 bash[28705]: 74982d89df8650da93103cd9126a631e562c24a212905c96273187f58f5045c3 2024-06-18T02:29:58.591 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:29:58 smithi001 systemd[1]: Started Ceph mon.smithi001 for 7eefb63e-2d1a-11ef-bc9d-c7b262605968. 2024-06-18T02:29:59.503 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout cluster: 2024-06-18T02:29:59.503 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout id: 7eefb63e-2d1a-11ef-bc9d-c7b262605968 2024-06-18T02:29:59.503 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-06-18T02:29:59.503 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 2024-06-18T02:29:59.503 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout services: 2024-06-18T02:29:59.503 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum smithi001 (age 0.551841s) 2024-06-18T02:29:59.503 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-06-18T02:29:59.503 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-06-18T02:29:59.503 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 2024-06-18T02:29:59.503 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout data: 2024-06-18T02:29:59.504 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-06-18T02:29:59.504 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-06-18T02:29:59.504 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-06-18T02:29:59.504 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout pgs: 2024-06-18T02:29:59.504 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 2024-06-18T02:29:59.504 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:58.942+0000 7f196d84a640 1 Processor -- start 2024-06-18T02:29:59.504 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:58.943+0000 7f196d84a640 1 -- start start 2024-06-18T02:29:59.504 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:58.943+0000 7f196d84a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1968077cd0 0x7f1968076250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:29:59.504 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:58.943+0000 7f196d84a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19680780e0 con 0x7f1968077cd0 2024-06-18T02:29:59.504 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:58.943+0000 7f1966ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1968077cd0 0x7f1968076250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:29:59.504 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:58.943+0000 7f1966ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1968077cd0 0x7f1968076250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38710/0 (socket says 172.21.15.1:38710) 2024-06-18T02:29:59.504 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:58.943+0000 7f1966ffd640 1 -- 172.21.15.1:0/2646973487 learned_addr learned my addr 172.21.15.1:0/2646973487 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:29:59.504 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:58.943+0000 7f1966ffd640 1 -- 172.21.15.1:0/2646973487 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1968076790 con 0x7f1968077cd0 2024-06-18T02:29:59.505 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:58.945+0000 7f1966ffd640 1 --2- 172.21.15.1:0/2646973487 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1968077cd0 0x7f1968076250 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f1958009840 tx=0x7f19580091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee50d3fc3fdb016e server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:29:59.505 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:58.946+0000 7f1965ffb640 1 -- 172.21.15.1:0/2646973487 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1958032440 con 0x7f1968077cd0 2024-06-18T02:29:59.505 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:58.946+0000 7f1965ffb640 1 -- 172.21.15.1:0/2646973487 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19580325a0 con 0x7f1968077cd0 2024-06-18T02:29:59.505 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:58.946+0000 7f1965ffb640 1 -- 172.21.15.1:0/2646973487 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f19580328d0 con 0x7f1968077cd0 2024-06-18T02:29:59.505 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:58.946+0000 7f196d84a640 1 -- 172.21.15.1:0/2646973487 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1968077cd0 msgr2=0x7f1968076250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:29:59.505 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:58.946+0000 7f196d84a640 1 --2- 172.21.15.1:0/2646973487 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1968077cd0 0x7f1968076250 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f1958009840 tx=0x7f19580091d0 comp rx=0 tx=0).stop 2024-06-18T02:29:59.505 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:58.946+0000 7f196d84a640 1 -- 172.21.15.1:0/2646973487 shutdown_connections 2024-06-18T02:29:59.505 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:58.946+0000 7f196d84a640 1 --2- 172.21.15.1:0/2646973487 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1968077cd0 0x7f1968076250 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:29:59.505 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:58.946+0000 7f196d84a640 1 -- 172.21.15.1:0/2646973487 >> 172.21.15.1:0/2646973487 conn(0x7f19680fd500 msgr2=0x7f19680ff910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:29:59.505 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:58.946+0000 7f196d84a640 1 -- 172.21.15.1:0/2646973487 shutdown_connections 2024-06-18T02:29:59.505 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:58.946+0000 7f196d84a640 1 -- 172.21.15.1:0/2646973487 wait complete. 2024-06-18T02:29:59.505 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:58.947+0000 7f196d84a640 1 Processor -- start 2024-06-18T02:29:59.505 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:58.947+0000 7f196d84a640 1 -- start start 2024-06-18T02:29:59.506 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:58.947+0000 7f196d84a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1968077cd0 0x7f1968112820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:29:59.506 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:58.947+0000 7f196d84a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1968112d60 con 0x7f1968077cd0 2024-06-18T02:29:59.506 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:58.947+0000 7f1966ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1968077cd0 0x7f1968112820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:29:59.506 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:58.947+0000 7f1966ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1968077cd0 0x7f1968112820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38722/0 (socket says 172.21.15.1:38722) 2024-06-18T02:29:59.506 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:58.947+0000 7f1966ffd640 1 -- 172.21.15.1:0/2522621235 learned_addr learned my addr 172.21.15.1:0/2522621235 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:29:59.506 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:58.948+0000 7f1966ffd640 1 -- 172.21.15.1:0/2522621235 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1958009450 con 0x7f1968077cd0 2024-06-18T02:29:59.506 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:58.948+0000 7f1966ffd640 1 --2- 172.21.15.1:0/2522621235 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1968077cd0 0x7f1968112820 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f195802a700 tx=0x7f1958032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:29:59.506 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:58.948+0000 7f194f7fe640 1 -- 172.21.15.1:0/2522621235 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1958031be0 con 0x7f1968077cd0 2024-06-18T02:29:59.506 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:58.948+0000 7f194f7fe640 1 -- 172.21.15.1:0/2522621235 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1958031d40 con 0x7f1968077cd0 2024-06-18T02:29:59.506 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:58.948+0000 7f196d84a640 1 -- 172.21.15.1:0/2522621235 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1968112f60 con 0x7f1968077cd0 2024-06-18T02:29:59.506 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:58.948+0000 7f194f7fe640 1 -- 172.21.15.1:0/2522621235 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f195803b5a0 con 0x7f1968077cd0 2024-06-18T02:29:59.506 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:58.948+0000 7f196d84a640 1 -- 172.21.15.1:0/2522621235 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1968113370 con 0x7f1968077cd0 2024-06-18T02:29:59.506 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:58.948+0000 7f194f7fe640 1 -- 172.21.15.1:0/2522621235 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f1958039040 con 0x7f1968077cd0 2024-06-18T02:29:59.506 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:58.948+0000 7f194f7fe640 1 -- 172.21.15.1:0/2522621235 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f1958041030 con 0x7f1968077cd0 2024-06-18T02:29:59.507 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:58.949+0000 7f196d84a640 1 -- 172.21.15.1:0/2522621235 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f196804b480 con 0x7f1968077cd0 2024-06-18T02:29:59.507 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:58.951+0000 7f194f7fe640 1 -- 172.21.15.1:0/2522621235 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71547 (secure 0 0 0) 0x7f1958037070 con 0x7f1968077cd0 2024-06-18T02:29:59.507 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:59.016+0000 7f196d84a640 1 -- 172.21.15.1:0/2522621235 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f1968076aa0 con 0x7f1968077cd0 2024-06-18T02:29:59.507 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:59.016+0000 7f194f7fe640 1 -- 172.21.15.1:0/2522621235 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7f195803bb80 con 0x7f1968077cd0 2024-06-18T02:29:59.507 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:59.017+0000 7f196d84a640 1 -- 172.21.15.1:0/2522621235 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1968077cd0 msgr2=0x7f1968112820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:29:59.507 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:59.017+0000 7f196d84a640 1 --2- 172.21.15.1:0/2522621235 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1968077cd0 0x7f1968112820 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f195802a700 tx=0x7f1958032000 comp rx=0 tx=0).stop 2024-06-18T02:29:59.507 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:59.017+0000 7f196d84a640 1 -- 172.21.15.1:0/2522621235 shutdown_connections 2024-06-18T02:29:59.507 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:59.017+0000 7f196d84a640 1 --2- 172.21.15.1:0/2522621235 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1968077cd0 0x7f1968112820 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:29:59.507 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:59.017+0000 7f196d84a640 1 -- 172.21.15.1:0/2522621235 >> 172.21.15.1:0/2522621235 conn(0x7f19680fd500 msgr2=0x7f19681093c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:29:59.507 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:59.017+0000 7f196d84a640 1 -- 172.21.15.1:0/2522621235 shutdown_connections 2024-06-18T02:29:59.507 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:59.017+0000 7f196d84a640 1 -- 172.21.15.1:0/2522621235 wait complete. 2024-06-18T02:29:59.507 INFO:teuthology.orchestra.run.smithi001.stdout:mon is available 2024-06-18T02:29:59.507 INFO:teuthology.orchestra.run.smithi001.stdout:Assimilating anything we can from ceph.conf... 2024-06-18T02:29:59.612 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:29:59 smithi001 ceph-mon[28724]: mon.smithi001 is new leader, mons smithi001 in quorum (ranks 0) 2024-06-18T02:29:59.612 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:29:59 smithi001 ceph-mon[28724]: monmap e1: 1 mons at {smithi001=[v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-06-18T02:29:59.612 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:29:59 smithi001 ceph-mon[28724]: fsmap 2024-06-18T02:29:59.612 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:29:59 smithi001 ceph-mon[28724]: osdmap e1: 0 total, 0 up, 0 in 2024-06-18T02:29:59.612 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:29:59 smithi001 ceph-mon[28724]: mgrmap e1: no daemons active 2024-06-18T02:29:59.612 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:29:59 smithi001 ceph-mon[28724]: from='client.? 172.21.15.1:0/2522621235' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-06-18T02:30:00.493 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 2024-06-18T02:30:00.493 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout [global] 2024-06-18T02:30:00.493 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout fsid = 7eefb63e-2d1a-11ef-bc9d-c7b262605968 2024-06-18T02:30:00.493 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.1:3300,v1:172.21.15.1:6789] 2024-06-18T02:30:00.493 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-06-18T02:30:00.493 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-06-18T02:30:00.493 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-06-18T02:30:00.494 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-06-18T02:30:00.494 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 2024-06-18T02:30:00.494 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout [mgr] 2024-06-18T02:30:00.494 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-06-18T02:30:00.494 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 2024-06-18T02:30:00.494 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout [osd] 2024-06-18T02:30:00.494 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-06-18T02:30:00.494 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-06-18T02:30:00.494 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-06-18T02:30:00.495 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-06-18T02:30:00.495 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-06-18T02:30:00.495 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:59.908+0000 7f2ac9f8e640 1 Processor -- start 2024-06-18T02:30:00.495 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:59.908+0000 7f2ac9f8e640 1 -- start start 2024-06-18T02:30:00.495 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:59.908+0000 7f2ac9f8e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ac4077cb0 0x7f2ac4076230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:00.495 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:59.908+0000 7f2ac9f8e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac40780c0 con 0x7f2ac4077cb0 2024-06-18T02:30:00.495 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:59.909+0000 7f2ac37fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ac4077cb0 0x7f2ac4076230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:30:00.495 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:59.909+0000 7f2ac37fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ac4077cb0 0x7f2ac4076230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38738/0 (socket says 172.21.15.1:38738) 2024-06-18T02:30:00.495 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:59.909+0000 7f2ac37fe640 1 -- 172.21.15.1:0/659125032 learned_addr learned my addr 172.21.15.1:0/659125032 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:30:00.495 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:59.909+0000 7f2ac37fe640 1 -- 172.21.15.1:0/659125032 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ac4076770 con 0x7f2ac4077cb0 2024-06-18T02:30:00.495 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:59.910+0000 7f2ac37fe640 1 --2- 172.21.15.1:0/659125032 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ac4077cb0 0x7f2ac4076230 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f2ab4009840 tx=0x7f2ab40091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abe5f9f6df6663ea server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:30:00.496 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:59.910+0000 7f2ac27fc640 1 -- 172.21.15.1:0/659125032 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ab4032440 con 0x7f2ac4077cb0 2024-06-18T02:30:00.496 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:59.910+0000 7f2ac27fc640 1 -- 172.21.15.1:0/659125032 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ab40325a0 con 0x7f2ac4077cb0 2024-06-18T02:30:00.496 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:59.910+0000 7f2ac27fc640 1 -- 172.21.15.1:0/659125032 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ab40328d0 con 0x7f2ac4077cb0 2024-06-18T02:30:00.496 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:59.910+0000 7f2ac9f8e640 1 -- 172.21.15.1:0/659125032 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ac4077cb0 msgr2=0x7f2ac4076230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:30:00.496 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:59.910+0000 7f2ac9f8e640 1 --2- 172.21.15.1:0/659125032 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ac4077cb0 0x7f2ac4076230 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f2ab4009840 tx=0x7f2ab40091d0 comp rx=0 tx=0).stop 2024-06-18T02:30:00.496 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:59.910+0000 7f2ac9f8e640 1 -- 172.21.15.1:0/659125032 shutdown_connections 2024-06-18T02:30:00.496 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:59.910+0000 7f2ac9f8e640 1 --2- 172.21.15.1:0/659125032 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ac4077cb0 0x7f2ac4076230 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:00.496 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:59.910+0000 7f2ac9f8e640 1 -- 172.21.15.1:0/659125032 >> 172.21.15.1:0/659125032 conn(0x7f2ac40fd500 msgr2=0x7f2ac40ff910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:30:00.496 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:59.910+0000 7f2ac9f8e640 1 -- 172.21.15.1:0/659125032 shutdown_connections 2024-06-18T02:30:00.496 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:59.910+0000 7f2ac9f8e640 1 -- 172.21.15.1:0/659125032 wait complete. 2024-06-18T02:30:00.496 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:59.911+0000 7f2ac9f8e640 1 Processor -- start 2024-06-18T02:30:00.496 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:59.911+0000 7f2ac9f8e640 1 -- start start 2024-06-18T02:30:00.496 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:59.911+0000 7f2ac9f8e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ac4077cb0 0x7f2ac4112830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:00.497 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:59.911+0000 7f2ac9f8e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac4112d70 con 0x7f2ac4077cb0 2024-06-18T02:30:00.497 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:59.911+0000 7f2ac37fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ac4077cb0 0x7f2ac4112830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:30:00.497 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:59.911+0000 7f2ac37fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ac4077cb0 0x7f2ac4112830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38740/0 (socket says 172.21.15.1:38740) 2024-06-18T02:30:00.497 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:59.911+0000 7f2ac37fe640 1 -- 172.21.15.1:0/3039905415 learned_addr learned my addr 172.21.15.1:0/3039905415 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:30:00.497 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:59.912+0000 7f2ac37fe640 1 -- 172.21.15.1:0/3039905415 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ab4009450 con 0x7f2ac4077cb0 2024-06-18T02:30:00.497 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:59.912+0000 7f2ac37fe640 1 --2- 172.21.15.1:0/3039905415 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ac4077cb0 0x7f2ac4112830 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f2ab402a700 tx=0x7f2ab4032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:30:00.497 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:59.912+0000 7f2aabfff640 1 -- 172.21.15.1:0/3039905415 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ab4031be0 con 0x7f2ac4077cb0 2024-06-18T02:30:00.497 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:59.912+0000 7f2aabfff640 1 -- 172.21.15.1:0/3039905415 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ab4031d40 con 0x7f2ac4077cb0 2024-06-18T02:30:00.497 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:59.912+0000 7f2ac9f8e640 1 -- 172.21.15.1:0/3039905415 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ac4112f70 con 0x7f2ac4077cb0 2024-06-18T02:30:00.497 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:59.912+0000 7f2ac9f8e640 1 -- 172.21.15.1:0/3039905415 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ac4113380 con 0x7f2ac4077cb0 2024-06-18T02:30:00.497 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:59.912+0000 7f2aabfff640 1 -- 172.21.15.1:0/3039905415 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ab403b5a0 con 0x7f2ac4077cb0 2024-06-18T02:30:00.497 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:59.912+0000 7f2aabfff640 1 -- 172.21.15.1:0/3039905415 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f2ab4039040 con 0x7f2ac4077cb0 2024-06-18T02:30:00.497 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:59.912+0000 7f2aabfff640 1 -- 172.21.15.1:0/3039905415 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f2ab4041030 con 0x7f2ac4077cb0 2024-06-18T02:30:00.498 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:59.913+0000 7f2ac9f8e640 1 -- 172.21.15.1:0/3039905415 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ac404b480 con 0x7f2ac4077cb0 2024-06-18T02:30:00.498 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:59.915+0000 7f2aabfff640 1 -- 172.21.15.1:0/3039905415 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71547 (secure 0 0 0) 0x7f2ab4037070 con 0x7f2ac4077cb0 2024-06-18T02:30:00.498 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:59.976+0000 7f2ac9f8e640 1 -- 172.21.15.1:0/3039905415 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f2ac4063ec0 con 0x7f2ac4077cb0 2024-06-18T02:30:00.498 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:59.979+0000 7f2aabfff640 1 -- 172.21.15.1:0/3039905415 <== mon.0 v2:172.21.15.1:3300/0 7 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f2ab4046c60 con 0x7f2ac4077cb0 2024-06-18T02:30:00.498 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:59.979+0000 7f2aabfff640 1 -- 172.21.15.1:0/3039905415 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+427 (secure 0 0 0) 0x7f2ab4054d60 con 0x7f2ac4077cb0 2024-06-18T02:30:00.498 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:59.980+0000 7f2ac9f8e640 1 -- 172.21.15.1:0/3039905415 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ac4077cb0 msgr2=0x7f2ac4112830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:30:00.498 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:59.980+0000 7f2ac9f8e640 1 --2- 172.21.15.1:0/3039905415 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ac4077cb0 0x7f2ac4112830 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f2ab402a700 tx=0x7f2ab4032000 comp rx=0 tx=0).stop 2024-06-18T02:30:00.498 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:59.980+0000 7f2ac9f8e640 1 -- 172.21.15.1:0/3039905415 shutdown_connections 2024-06-18T02:30:00.498 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:59.980+0000 7f2ac9f8e640 1 --2- 172.21.15.1:0/3039905415 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ac4077cb0 0x7f2ac4112830 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:00.498 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:59.980+0000 7f2ac9f8e640 1 -- 172.21.15.1:0/3039905415 >> 172.21.15.1:0/3039905415 conn(0x7f2ac40fd500 msgr2=0x7f2ac4109400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:30:00.498 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:59.980+0000 7f2ac9f8e640 1 -- 172.21.15.1:0/3039905415 shutdown_connections 2024-06-18T02:30:00.499 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:29:59.980+0000 7f2ac9f8e640 1 -- 172.21.15.1:0/3039905415 wait complete. 2024-06-18T02:30:00.499 INFO:teuthology.orchestra.run.smithi001.stdout:Generating new minimal ceph.conf... 2024-06-18T02:30:00.722 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:00 smithi001 ceph-mon[28724]: from='client.? 172.21.15.1:0/3039905415' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-06-18T02:30:00.722 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:00 smithi001 ceph-mon[28724]: from='client.? 172.21.15.1:0/3039905415' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-06-18T02:30:01.497 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:00.888+0000 7f5c03c24640 1 Processor -- start 2024-06-18T02:30:01.497 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:00.888+0000 7f5c03c24640 1 -- start start 2024-06-18T02:30:01.497 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:00.888+0000 7f5c03c24640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5bfc100c90 0x7f5bfc1010a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:01.497 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:00.888+0000 7f5c03c24640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bfc1015e0 con 0x7f5bfc100c90 2024-06-18T02:30:01.497 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:00.889+0000 7f5c019c0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5bfc100c90 0x7f5bfc1010a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:30:01.497 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:00.889+0000 7f5c019c0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5bfc100c90 0x7f5bfc1010a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38748/0 (socket says 172.21.15.1:38748) 2024-06-18T02:30:01.498 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:00.889+0000 7f5c019c0640 1 -- 172.21.15.1:0/231041840 learned_addr learned my addr 172.21.15.1:0/231041840 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:30:01.498 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:00.889+0000 7f5c019c0640 1 -- 172.21.15.1:0/231041840 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bfc101720 con 0x7f5bfc100c90 2024-06-18T02:30:01.498 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:00.890+0000 7f5c019c0640 1 --2- 172.21.15.1:0/231041840 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5bfc100c90 0x7f5bfc1010a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f5bf8009840 tx=0x7f5bf80091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba1c938eebc7a06e server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:30:01.498 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:00.890+0000 7f5c009be640 1 -- 172.21.15.1:0/231041840 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5bf8032440 con 0x7f5bfc100c90 2024-06-18T02:30:01.498 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:00.890+0000 7f5c009be640 1 -- 172.21.15.1:0/231041840 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f5bf8032a30 con 0x7f5bfc100c90 2024-06-18T02:30:01.498 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:00.890+0000 7f5c009be640 1 -- 172.21.15.1:0/231041840 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5bf8031ad0 con 0x7f5bfc100c90 2024-06-18T02:30:01.498 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:00.890+0000 7f5c03c24640 1 -- 172.21.15.1:0/231041840 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5bfc100c90 msgr2=0x7f5bfc1010a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:30:01.499 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:00.890+0000 7f5c03c24640 1 --2- 172.21.15.1:0/231041840 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5bfc100c90 0x7f5bfc1010a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f5bf8009840 tx=0x7f5bf80091d0 comp rx=0 tx=0).stop 2024-06-18T02:30:01.499 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:00.890+0000 7f5c03c24640 1 -- 172.21.15.1:0/231041840 shutdown_connections 2024-06-18T02:30:01.499 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:00.890+0000 7f5c03c24640 1 --2- 172.21.15.1:0/231041840 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5bfc100c90 0x7f5bfc1010a0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:01.499 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:00.890+0000 7f5c03c24640 1 -- 172.21.15.1:0/231041840 >> 172.21.15.1:0/231041840 conn(0x7f5bfc0fc450 msgr2=0x7f5bfc0fe880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:30:01.499 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:00.890+0000 7f5c03c24640 1 -- 172.21.15.1:0/231041840 shutdown_connections 2024-06-18T02:30:01.499 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:00.891+0000 7f5c03c24640 1 -- 172.21.15.1:0/231041840 wait complete. 2024-06-18T02:30:01.499 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:00.891+0000 7f5c03c24640 1 Processor -- start 2024-06-18T02:30:01.499 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:00.891+0000 7f5c03c24640 1 -- start start 2024-06-18T02:30:01.499 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:00.891+0000 7f5c03c24640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5bfc100c90 0x7f5bfc106e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:01.499 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:00.891+0000 7f5c03c24640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bfc1073a0 con 0x7f5bfc100c90 2024-06-18T02:30:01.499 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:00.891+0000 7f5c019c0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5bfc100c90 0x7f5bfc106e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:30:01.500 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:00.891+0000 7f5c019c0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5bfc100c90 0x7f5bfc106e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38762/0 (socket says 172.21.15.1:38762) 2024-06-18T02:30:01.500 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:00.891+0000 7f5c019c0640 1 -- 172.21.15.1:0/1064619407 learned_addr learned my addr 172.21.15.1:0/1064619407 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:30:01.500 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:00.892+0000 7f5c019c0640 1 -- 172.21.15.1:0/1064619407 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bf8009450 con 0x7f5bfc100c90 2024-06-18T02:30:01.500 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:00.892+0000 7f5c019c0640 1 --2- 172.21.15.1:0/1064619407 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5bfc100c90 0x7f5bfc106e60 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f5bf8009420 tx=0x7f5bf8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:30:01.500 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:00.892+0000 7f5bf27fc640 1 -- 172.21.15.1:0/1064619407 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5bf803d5b0 con 0x7f5bfc100c90 2024-06-18T02:30:01.500 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:00.892+0000 7f5bf27fc640 1 -- 172.21.15.1:0/1064619407 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f5bf803dba0 con 0x7f5bfc100c90 2024-06-18T02:30:01.500 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:00.892+0000 7f5c03c24640 1 -- 172.21.15.1:0/1064619407 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bfc105430 con 0x7f5bfc100c90 2024-06-18T02:30:01.500 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:00.892+0000 7f5bf27fc640 1 -- 172.21.15.1:0/1064619407 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5bf803bb40 con 0x7f5bfc100c90 2024-06-18T02:30:01.500 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:00.892+0000 7f5c03c24640 1 -- 172.21.15.1:0/1064619407 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bfc1058a0 con 0x7f5bfc100c90 2024-06-18T02:30:01.500 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:00.892+0000 7f5bf27fc640 1 -- 172.21.15.1:0/1064619407 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f5bf8039070 con 0x7f5bfc100c90 2024-06-18T02:30:01.500 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:00.892+0000 7f5bf27fc640 1 -- 172.21.15.1:0/1064619407 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f5bf8046900 con 0x7f5bfc100c90 2024-06-18T02:30:01.500 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:00.893+0000 7f5c03c24640 1 -- 172.21.15.1:0/1064619407 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bfc05ead0 con 0x7f5bfc100c90 2024-06-18T02:30:01.500 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:00.895+0000 7f5bf27fc640 1 -- 172.21.15.1:0/1064619407 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71547 (secure 0 0 0) 0x7f5bf8041020 con 0x7f5bfc100c90 2024-06-18T02:30:01.501 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:00.956+0000 7f5c03c24640 1 -- 172.21.15.1:0/1064619407 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f5bfc1029b0 con 0x7f5bfc100c90 2024-06-18T02:30:01.501 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:00.957+0000 7f5bf27fc640 1 -- 172.21.15.1:0/1064619407 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+173 (secure 0 0 0) 0x7f5bf8037070 con 0x7f5bfc100c90 2024-06-18T02:30:01.501 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:00.957+0000 7f5c03c24640 1 -- 172.21.15.1:0/1064619407 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5bfc100c90 msgr2=0x7f5bfc106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:30:01.501 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:00.957+0000 7f5c03c24640 1 --2- 172.21.15.1:0/1064619407 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5bfc100c90 0x7f5bfc106e60 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f5bf8009420 tx=0x7f5bf8008040 comp rx=0 tx=0).stop 2024-06-18T02:30:01.501 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:00.957+0000 7f5c03c24640 1 -- 172.21.15.1:0/1064619407 shutdown_connections 2024-06-18T02:30:01.501 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:00.957+0000 7f5c03c24640 1 --2- 172.21.15.1:0/1064619407 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5bfc100c90 0x7f5bfc106e60 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:01.501 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:00.957+0000 7f5c03c24640 1 -- 172.21.15.1:0/1064619407 >> 172.21.15.1:0/1064619407 conn(0x7f5bfc0fc450 msgr2=0x7f5bfc0fd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:30:01.501 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:00.958+0000 7f5c03c24640 1 -- 172.21.15.1:0/1064619407 shutdown_connections 2024-06-18T02:30:01.501 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:00.958+0000 7f5c03c24640 1 -- 172.21.15.1:0/1064619407 wait complete. 2024-06-18T02:30:01.501 INFO:teuthology.orchestra.run.smithi001.stdout:Restarting the monitor... 2024-06-18T02:30:01.517 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:01 smithi001 ceph-mon[28724]: from='client.? 172.21.15.1:0/1064619407' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:30:01.769 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:01 smithi001 systemd[1]: Stopping Ceph mon.smithi001 for 7eefb63e-2d1a-11ef-bc9d-c7b262605968... 2024-06-18T02:30:02.101 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:01 smithi001 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-mon-smithi001[28720]: 2024-06-18T02:30:01.768+0000 7f220c447640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi001 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-06-18T02:30:02.101 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:01 smithi001 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-mon-smithi001[28720]: 2024-06-18T02:30:01.768+0000 7f220c447640 -1 mon.smithi001@0(leader) e1 *** Got Signal Terminated *** 2024-06-18T02:30:02.101 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:01 smithi001 podman[28966]: 2024-06-18 02:30:01.913112417 +0000 UTC m=+0.244292616 container died 74982d89df8650da93103cd9126a631e562c24a212905c96273187f58f5045c3 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-mon-smithi001, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, RELEASE=quincy-3fe4105, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240612, org.label-schema.license=GPLv2, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-06-18T02:30:02.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 podman[28966]: 2024-06-18 02:30:02.201971488 +0000 UTC m=+0.533151694 container cleanup 74982d89df8650da93103cd9126a631e562c24a212905c96273187f58f5045c3 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-mon-smithi001, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, RELEASE=quincy-3fe4105, org.label-schema.build-date=20240612, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-18T02:30:02.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 bash[28966]: ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-mon-smithi001 2024-06-18T02:30:02.761 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 podman[28978]: 2024-06-18 02:30:02.411329742 +0000 UTC m=+0.500326629 container remove 74982d89df8650da93103cd9126a631e562c24a212905c96273187f58f5045c3 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-mon-smithi001, RELEASE=quincy-3fe4105, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240612, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-06-18T02:30:02.761 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 systemd[1]: ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968@mon.smithi001.service: Deactivated successfully. 2024-06-18T02:30:02.761 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 systemd[1]: Stopped Ceph mon.smithi001 for 7eefb63e-2d1a-11ef-bc9d-c7b262605968. 2024-06-18T02:30:02.761 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 systemd[1]: Starting Ceph mon.smithi001 for 7eefb63e-2d1a-11ef-bc9d-c7b262605968... 2024-06-18T02:30:03.030 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 podman[29079]: 2024-06-18 02:30:02.762290495 +0000 UTC m=+0.096699572 container create e9c952cac6b572f50bdf1d3bca79e77ed10b21ff653534572a615000cb45c2eb (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-mon-smithi001, GIT_CLEAN=True, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240612, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=quincy-3fe4105, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-06-18T02:30:03.030 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 podman[29079]: 2024-06-18 02:30:02.683086512 +0000 UTC m=+0.017495548 image pull 1a271607156e1b6becab587233eefa647399ff75aa2bb76fdb2f0360ec5b7818 quay.ceph.io/ceph-ci/ceph:quincy 2024-06-18T02:30:03.030 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 podman[29079]: 2024-06-18 02:30:02.956210634 +0000 UTC m=+0.290619670 container init e9c952cac6b572f50bdf1d3bca79e77ed10b21ff653534572a615000cb45c2eb (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-mon-smithi001, org.label-schema.license=GPLv2, RELEASE=quincy-3fe4105, maintainer=Guillaume Abrioux , GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240612, CEPH_POINT_RELEASE=) 2024-06-18T02:30:03.030 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 podman[29079]: 2024-06-18 02:30:02.959333213 +0000 UTC m=+0.293742239 container start e9c952cac6b572f50bdf1d3bca79e77ed10b21ff653534572a615000cb45c2eb (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-mon-smithi001, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240612, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=quincy-3fe4105, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-06-18T02:30:03.030 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: set uid:gid to 167:167 (ceph:ceph) 2024-06-18T02:30:03.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: ceph version 17.2.7-1012-g3fe41055 (3fe410552b224c0fbe57dfbac3e18568c0f8fef6) quincy (stable), process ceph-mon, pid 2 2024-06-18T02:30:03.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: pidfile_write: ignore empty --pid-file 2024-06-18T02:30:03.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: load: jerasure load: lrc 2024-06-18T02:30:03.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: RocksDB version: 6.15.5 2024-06-18T02:30:03.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-06-18T02:30:03.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Compile date Jun 17 2024 2024-06-18T02:30:03.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: DB SUMMARY 2024-06-18T02:30:03.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: DB Session ID: 3B1XUDE1CX31YX7Q94WM 2024-06-18T02:30:03.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: CURRENT file: CURRENT 2024-06-18T02:30:03.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: IDENTITY file: IDENTITY 2024-06-18T02:30:03.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-06-18T02:30:03.031 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi001/store.db dir, Total Num: 1, files: 000008.sst 2024-06-18T02:30:03.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi001/store.db: 000010.log size: 98140 ; 2024-06-18T02:30:03.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.error_if_exists: 0 2024-06-18T02:30:03.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.create_if_missing: 0 2024-06-18T02:30:03.032 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.paranoid_checks: 1 2024-06-18T02:30:03.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-06-18T02:30:03.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.env: 0x555c7d10f440 2024-06-18T02:30:03.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.fs: Posix File System 2024-06-18T02:30:03.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.info_log: 0x555c7e0ed140 2024-06-18T02:30:03.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.max_file_opening_threads: 16 2024-06-18T02:30:03.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.statistics: (nil) 2024-06-18T02:30:03.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.use_fsync: 0 2024-06-18T02:30:03.033 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.max_log_file_size: 0 2024-06-18T02:30:03.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-06-18T02:30:03.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.log_file_time_to_roll: 0 2024-06-18T02:30:03.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.keep_log_file_num: 1000 2024-06-18T02:30:03.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.recycle_log_file_num: 0 2024-06-18T02:30:03.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.allow_fallocate: 1 2024-06-18T02:30:03.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.allow_mmap_reads: 0 2024-06-18T02:30:03.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.allow_mmap_writes: 0 2024-06-18T02:30:03.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.use_direct_reads: 0 2024-06-18T02:30:03.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-06-18T02:30:03.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.create_missing_column_families: 0 2024-06-18T02:30:03.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.db_log_dir: 2024-06-18T02:30:03.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi001/store.db 2024-06-18T02:30:03.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.table_cache_numshardbits: 6 2024-06-18T02:30:03.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.WAL_ttl_seconds: 0 2024-06-18T02:30:03.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.WAL_size_limit_MB: 0 2024-06-18T02:30:03.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-06-18T02:30:03.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-06-18T02:30:03.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.is_fd_close_on_exec: 1 2024-06-18T02:30:03.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.advise_random_on_open: 1 2024-06-18T02:30:03.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.db_write_buffer_size: 0 2024-06-18T02:30:03.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.write_buffer_manager: 0x555c7e1aa840 2024-06-18T02:30:03.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-06-18T02:30:03.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-06-18T02:30:03.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-06-18T02:30:03.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.use_adaptive_mutex: 0 2024-06-18T02:30:03.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.rate_limiter: (nil) 2024-06-18T02:30:03.035 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-06-18T02:30:03.036 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.wal_recovery_mode: 2 2024-06-18T02:30:03.036 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.enable_thread_tracking: 0 2024-06-18T02:30:03.036 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.enable_pipelined_write: 0 2024-06-18T02:30:03.036 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.unordered_write: 0 2024-06-18T02:30:03.036 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-06-18T02:30:03.036 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-06-18T02:30:03.036 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-06-18T02:30:03.036 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-06-18T02:30:03.036 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.row_cache: None 2024-06-18T02:30:03.036 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.wal_filter: None 2024-06-18T02:30:03.036 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-06-18T02:30:03.036 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.allow_ingest_behind: 0 2024-06-18T02:30:03.036 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.preserve_deletes: 0 2024-06-18T02:30:03.036 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.two_write_queues: 0 2024-06-18T02:30:03.036 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.manual_wal_flush: 0 2024-06-18T02:30:03.037 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.atomic_flush: 0 2024-06-18T02:30:03.037 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-06-18T02:30:03.037 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.persist_stats_to_disk: 0 2024-06-18T02:30:03.037 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.write_dbid_to_manifest: 0 2024-06-18T02:30:03.037 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.log_readahead_size: 0 2024-06-18T02:30:03.037 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-06-18T02:30:03.037 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.best_efforts_recovery: 0 2024-06-18T02:30:03.037 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-06-18T02:30:03.037 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-06-18T02:30:03.037 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.allow_data_in_errors: 0 2024-06-18T02:30:03.037 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.db_host_id: __hostname__ 2024-06-18T02:30:03.037 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.max_background_jobs: 2 2024-06-18T02:30:03.037 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.max_background_compactions: -1 2024-06-18T02:30:03.037 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.max_subcompactions: 1 2024-06-18T02:30:03.037 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-06-18T02:30:03.038 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-06-18T02:30:03.038 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.delayed_write_rate : 16777216 2024-06-18T02:30:03.038 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.max_total_wal_size: 0 2024-06-18T02:30:03.038 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-06-18T02:30:03.038 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.stats_dump_period_sec: 600 2024-06-18T02:30:03.038 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.stats_persist_period_sec: 600 2024-06-18T02:30:03.038 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-06-18T02:30:03.038 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.max_open_files: -1 2024-06-18T02:30:03.038 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.bytes_per_sync: 0 2024-06-18T02:30:03.038 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.wal_bytes_per_sync: 0 2024-06-18T02:30:03.038 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.strict_bytes_per_sync: 0 2024-06-18T02:30:03.038 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.compaction_readahead_size: 0 2024-06-18T02:30:03.039 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.max_background_flushes: -1 2024-06-18T02:30:03.039 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Compression algorithms supported: 2024-06-18T02:30:03.039 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: kZSTD supported: 0 2024-06-18T02:30:03.039 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: kXpressCompression supported: 0 2024-06-18T02:30:03.039 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: kBZip2Compression supported: 0 2024-06-18T02:30:03.039 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-06-18T02:30:03.039 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: kLZ4Compression supported: 1 2024-06-18T02:30:03.039 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: kZlibCompression supported: 1 2024-06-18T02:30:03.039 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: kLZ4HCCompression supported: 1 2024-06-18T02:30:03.039 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: kSnappyCompression supported: 1 2024-06-18T02:30:03.039 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Fast CRC32 supported: Supported on x86 2024-06-18T02:30:03.039 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi001/store.db/MANIFEST-000009 2024-06-18T02:30:03.039 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-06-18T02:30:03.039 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-06-18T02:30:03.040 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.merge_operator: 2024-06-18T02:30:03.040 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.compaction_filter: None 2024-06-18T02:30:03.040 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.compaction_filter_factory: None 2024-06-18T02:30:03.040 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.sst_partitioner_factory: None 2024-06-18T02:30:03.040 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.memtable_factory: SkipListFactory 2024-06-18T02:30:03.040 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.table_factory: BlockBasedTable 2024-06-18T02:30:03.040 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x555c7e093a68) 2024-06-18T02:30:03.040 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: cache_index_and_filter_blocks: 1 2024-06-18T02:30:03.040 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-06-18T02:30:03.040 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-06-18T02:30:03.040 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: pin_top_level_index_and_filter: 1 2024-06-18T02:30:03.040 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: index_type: 0 2024-06-18T02:30:03.040 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: data_block_index_type: 0 2024-06-18T02:30:03.040 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: index_shortening: 1 2024-06-18T02:30:03.041 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: data_block_hash_table_util_ratio: 0.750000 2024-06-18T02:30:03.041 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: hash_index_allow_collision: 1 2024-06-18T02:30:03.041 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: checksum: 1 2024-06-18T02:30:03.041 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: no_block_cache: 0 2024-06-18T02:30:03.041 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: block_cache: 0x555c7e0c91f0 2024-06-18T02:30:03.041 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: block_cache_name: BinnedLRUCache 2024-06-18T02:30:03.041 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: block_cache_options: 2024-06-18T02:30:03.041 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: capacity : 536870912 2024-06-18T02:30:03.041 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: num_shard_bits : 4 2024-06-18T02:30:03.041 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: strict_capacity_limit : 0 2024-06-18T02:30:03.041 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: high_pri_pool_ratio: 0.000 2024-06-18T02:30:03.042 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: block_cache_compressed: (nil) 2024-06-18T02:30:03.042 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: persistent_cache: (nil) 2024-06-18T02:30:03.042 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: block_size: 4096 2024-06-18T02:30:03.042 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: block_size_deviation: 10 2024-06-18T02:30:03.042 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: block_restart_interval: 16 2024-06-18T02:30:03.042 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: index_block_restart_interval: 1 2024-06-18T02:30:03.042 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: metadata_block_size: 4096 2024-06-18T02:30:03.042 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: partition_filters: 0 2024-06-18T02:30:03.042 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: use_delta_encoding: 1 2024-06-18T02:30:03.042 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-06-18T02:30:03.042 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: whole_key_filtering: 1 2024-06-18T02:30:03.043 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: verify_compression: 0 2024-06-18T02:30:03.043 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: read_amp_bytes_per_bit: 0 2024-06-18T02:30:03.043 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: format_version: 4 2024-06-18T02:30:03.043 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: enable_index_compression: 1 2024-06-18T02:30:03.043 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: block_align: 0 2024-06-18T02:30:03.043 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.write_buffer_size: 33554432 2024-06-18T02:30:03.043 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.max_write_buffer_number: 2 2024-06-18T02:30:03.043 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.compression: NoCompression 2024-06-18T02:30:03.043 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.bottommost_compression: Disabled 2024-06-18T02:30:03.043 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.prefix_extractor: nullptr 2024-06-18T02:30:03.043 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-06-18T02:30:03.043 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.num_levels: 7 2024-06-18T02:30:03.043 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-06-18T02:30:03.043 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-06-18T02:30:03.044 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-06-18T02:30:03.044 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-06-18T02:30:03.044 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-06-18T02:30:03.044 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-06-18T02:30:03.044 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-06-18T02:30:03.044 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-06-18T02:30:03.044 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-06-18T02:30:03.044 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-06-18T02:30:03.044 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.compression_opts.window_bits: -14 2024-06-18T02:30:03.044 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.compression_opts.level: 32767 2024-06-18T02:30:03.044 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.compression_opts.strategy: 0 2024-06-18T02:30:03.044 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-06-18T02:30:03.044 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-06-18T02:30:03.045 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-06-18T02:30:03.045 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.compression_opts.enabled: false 2024-06-18T02:30:03.045 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-06-18T02:30:03.045 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-06-18T02:30:03.045 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-06-18T02:30:03.045 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.target_file_size_base: 67108864 2024-06-18T02:30:03.045 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.target_file_size_multiplier: 1 2024-06-18T02:30:03.045 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-06-18T02:30:03.045 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-06-18T02:30:03.045 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-06-18T02:30:03.045 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-06-18T02:30:03.045 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-06-18T02:30:03.045 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-06-18T02:30:03.046 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-06-18T02:30:03.046 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-06-18T02:30:03.046 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-06-18T02:30:03.046 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-06-18T02:30:03.046 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-06-18T02:30:03.046 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-06-18T02:30:03.046 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.arena_block_size: 4194304 2024-06-18T02:30:03.046 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-06-18T02:30:03.046 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-06-18T02:30:03.046 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-06-18T02:30:03.046 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.disable_auto_compactions: 0 2024-06-18T02:30:03.046 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-06-18T02:30:03.046 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-06-18T02:30:03.047 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-06-18T02:30:03.047 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-06-18T02:30:03.047 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-06-18T02:30:03.047 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-06-18T02:30:03.047 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-06-18T02:30:03.047 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-06-18T02:30:03.047 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-06-18T02:30:03.047 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-06-18T02:30:03.047 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.table_properties_collectors: 2024-06-18T02:30:03.047 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.inplace_update_support: 0 2024-06-18T02:30:03.047 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.inplace_update_num_locks: 10000 2024-06-18T02:30:03.047 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-06-18T02:30:03.047 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-06-18T02:30:03.048 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.memtable_huge_page_size: 0 2024-06-18T02:30:03.048 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.bloom_locality: 0 2024-06-18T02:30:03.048 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.max_successive_merges: 0 2024-06-18T02:30:03.048 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.optimize_filters_for_hits: 0 2024-06-18T02:30:03.048 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.paranoid_file_checks: 0 2024-06-18T02:30:03.048 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.force_consistency_checks: 1 2024-06-18T02:30:03.048 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.report_bg_io_stats: 0 2024-06-18T02:30:03.048 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.ttl: 2592000 2024-06-18T02:30:03.048 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.periodic_compaction_seconds: 0 2024-06-18T02:30:03.048 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.enable_blob_files: false 2024-06-18T02:30:03.048 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.min_blob_size: 0 2024-06-18T02:30:03.048 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.blob_file_size: 268435456 2024-06-18T02:30:03.049 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.blob_compression_type: NoCompression 2024-06-18T02:30:03.049 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.enable_blob_garbage_collection: false 2024-06-18T02:30:03.049 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-06-18T02:30:03.049 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi001/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-06-18T02:30:03.049 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 5 2024-06-18T02:30:03.050 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: [db/version_set.cc:4082] Creating manifest 13 2024-06-18T02:30:03.050 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: EVENT_LOG_v1 {"time_micros": 1718677802982834, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-06-18T02:30:03.050 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #10 mode 2 2024-06-18T02:30:03.050 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-06-18T02:30:03.050 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: EVENT_LOG_v1 {"time_micros": 1718677802983743, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 93346, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 91458, "index_size": 248, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13576, "raw_average_key_size": 49, "raw_value_size": 84022, "raw_average_value_size": 307, "num_data_blocks": 11, "num_entries": 273, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1718677802, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "4035d54a-d814-4b9f-95a9-a4511e6a1cf1", "db_session_id": "3B1XUDE1CX31YX7Q94WM"}} 2024-06-18T02:30:03.050 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: [db/version_set.cc:4082] Creating manifest 15 2024-06-18T02:30:03.050 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: EVENT_LOG_v1 {"time_micros": 1718677802984092, "job": 1, "event": "recovery_finished"} 2024-06-18T02:30:03.050 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-smithi001/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-06-18T02:30:03.050 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x555c7e0fc700 2024-06-18T02:30:03.050 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: DB pointer 0x555c7e0cc000 2024-06-18T02:30:03.050 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-06-18T02:30:03.050 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-06-18T02:30:03.050 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: ** DB Stats ** 2024-06-18T02:30:03.051 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-18T02:30:03.051 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-06-18T02:30:03.051 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-06-18T02:30:03.051 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-06-18T02:30:03.051 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-06-18T02:30:03.051 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-06-18T02:30:03.051 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-06-18T02:30:03.051 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: 2024-06-18T02:30:03.051 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: ** Compaction Stats [default] ** 2024-06-18T02:30:03.051 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-06-18T02:30:03.051 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-06-18T02:30:03.051 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: L0 2/0 92.96 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 146.9 0.00 0.00 1 0.001 0 0 2024-06-18T02:30:03.051 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Sum 2/0 92.96 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 146.9 0.00 0.00 1 0.001 0 0 2024-06-18T02:30:03.052 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 146.9 0.00 0.00 1 0.001 0 0 2024-06-18T02:30:03.052 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: 2024-06-18T02:30:03.052 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: ** Compaction Stats [default] ** 2024-06-18T02:30:03.052 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-06-18T02:30:03.052 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-06-18T02:30:03.052 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 146.9 0.00 0.00 1 0.001 0 0 2024-06-18T02:30:03.052 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-18T02:30:03.052 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-06-18T02:30:03.052 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-06-18T02:30:03.052 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-06-18T02:30:03.052 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-06-18T02:30:03.052 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: AddFile(Keys): cumulative 0, interval 0 2024-06-18T02:30:03.052 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Cumulative compaction: 0.00 GB write, 27.33 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-18T02:30:03.052 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Interval compaction: 0.00 GB write, 27.33 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-18T02:30:03.053 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-06-18T02:30:03.053 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: 2024-06-18T02:30:03.053 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: ** File Read Latency Histogram By Level [default] ** 2024-06-18T02:30:03.053 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: 2024-06-18T02:30:03.053 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: ** Compaction Stats [default] ** 2024-06-18T02:30:03.053 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-06-18T02:30:03.053 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-06-18T02:30:03.053 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: L0 2/0 92.96 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 146.9 0.00 0.00 1 0.001 0 0 2024-06-18T02:30:03.053 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Sum 2/0 92.96 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 146.9 0.00 0.00 1 0.001 0 0 2024-06-18T02:30:03.053 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-06-18T02:30:03.053 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: 2024-06-18T02:30:03.053 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: ** Compaction Stats [default] ** 2024-06-18T02:30:03.053 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-06-18T02:30:03.054 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-06-18T02:30:03.054 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 146.9 0.00 0.00 1 0.001 0 0 2024-06-18T02:30:03.054 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-18T02:30:03.054 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-06-18T02:30:03.054 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-06-18T02:30:03.054 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-06-18T02:30:03.054 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-06-18T02:30:03.054 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: AddFile(Keys): cumulative 0, interval 0 2024-06-18T02:30:03.054 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Cumulative compaction: 0.00 GB write, 27.07 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-18T02:30:03.054 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-18T02:30:03.054 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-06-18T02:30:03.054 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: 2024-06-18T02:30:03.054 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: ** File Read Latency Histogram By Level [default] ** 2024-06-18T02:30:03.055 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: starting mon.smithi001 rank 0 at public addrs [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] at bind addrs [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi001 fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 2024-06-18T02:30:03.055 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: mon.smithi001@-1(???) e1 preinit fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 2024-06-18T02:30:03.055 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: mon.smithi001@-1(???).mds e1 new map 2024-06-18T02:30:03.055 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: mon.smithi001@-1(???).mds e1 print_map 2024-06-18T02:30:03.055 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: e1 2024-06-18T02:30:03.055 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-06-18T02:30:03.055 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-06-18T02:30:03.055 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: legacy client fscid: -1 2024-06-18T02:30:03.055 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: 2024-06-18T02:30:03.055 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: No filesystems configured 2024-06-18T02:30:03.055 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: mon.smithi001@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-06-18T02:30:03.055 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: mon.smithi001@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-06-18T02:30:03.055 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: mon.smithi001@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-06-18T02:30:03.055 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: mon.smithi001@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-06-18T02:30:03.056 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:02 smithi001 ceph-mon[29099]: mon.smithi001@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-06-18T02:30:03.056 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:03 smithi001 bash[29079]: e9c952cac6b572f50bdf1d3bca79e77ed10b21ff653534572a615000cb45c2eb 2024-06-18T02:30:03.056 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:03 smithi001 systemd[1]: Started Ceph mon.smithi001 for 7eefb63e-2d1a-11ef-bc9d-c7b262605968. 2024-06-18T02:30:03.056 INFO:teuthology.orchestra.run.smithi001.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-06-18T02:30:03.316 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:03 smithi001 ceph-mon[29099]: mon.smithi001 is new leader, mons smithi001 in quorum (ranks 0) 2024-06-18T02:30:03.316 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:03 smithi001 ceph-mon[29099]: monmap e1: 1 mons at {smithi001=[v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-06-18T02:30:03.316 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:03 smithi001 ceph-mon[29099]: fsmap 2024-06-18T02:30:03.316 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:03 smithi001 ceph-mon[29099]: osdmap e1: 0 total, 0 up, 0 in 2024-06-18T02:30:03.316 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:03 smithi001 ceph-mon[29099]: mgrmap e1: no daemons active 2024-06-18T02:30:04.025 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:03.396+0000 7f84836b9640 1 Processor -- start 2024-06-18T02:30:04.025 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:03.397+0000 7f84836b9640 1 -- start start 2024-06-18T02:30:04.025 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:03.397+0000 7f84836b9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f847c075be0 0x7f847c104950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:04.025 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:03.397+0000 7f84836b9640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f847c104e90 con 0x7f847c075be0 2024-06-18T02:30:04.026 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:03.397+0000 7f8481455640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f847c075be0 0x7f847c104950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:30:04.026 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:03.397+0000 7f8481455640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f847c075be0 0x7f847c104950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38772/0 (socket says 172.21.15.1:38772) 2024-06-18T02:30:04.026 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:03.397+0000 7f8481455640 1 -- 172.21.15.1:0/1959351431 learned_addr learned my addr 172.21.15.1:0/1959351431 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:30:04.026 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:03.398+0000 7f8481455640 1 -- 172.21.15.1:0/1959351431 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f847c104fd0 con 0x7f847c075be0 2024-06-18T02:30:04.026 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:03.399+0000 7f8481455640 1 --2- 172.21.15.1:0/1959351431 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f847c075be0 0x7f847c104950 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f8478009840 tx=0x7f84780091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0089b4f49fa5bd5 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:30:04.026 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:03.399+0000 7f8473fff640 1 -- 172.21.15.1:0/1959351431 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8478032440 con 0x7f847c075be0 2024-06-18T02:30:04.026 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:03.400+0000 7f8473fff640 1 -- 172.21.15.1:0/1959351431 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f8478032a30 con 0x7f847c075be0 2024-06-18T02:30:04.026 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:03.400+0000 7f8473fff640 1 -- 172.21.15.1:0/1959351431 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8478031ad0 con 0x7f847c075be0 2024-06-18T02:30:04.027 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:03.400+0000 7f84836b9640 1 -- 172.21.15.1:0/1959351431 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f847c075be0 msgr2=0x7f847c104950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:30:04.027 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:03.400+0000 7f84836b9640 1 --2- 172.21.15.1:0/1959351431 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f847c075be0 0x7f847c104950 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f8478009840 tx=0x7f84780091d0 comp rx=0 tx=0).stop 2024-06-18T02:30:04.027 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:03.400+0000 7f84836b9640 1 -- 172.21.15.1:0/1959351431 shutdown_connections 2024-06-18T02:30:04.027 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:03.400+0000 7f84836b9640 1 --2- 172.21.15.1:0/1959351431 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f847c075be0 0x7f847c104950 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:04.027 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:03.400+0000 7f84836b9640 1 -- 172.21.15.1:0/1959351431 >> 172.21.15.1:0/1959351431 conn(0x7f847c0fc480 msgr2=0x7f847c0fe890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:30:04.027 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:03.400+0000 7f84836b9640 1 -- 172.21.15.1:0/1959351431 shutdown_connections 2024-06-18T02:30:04.027 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:03.400+0000 7f84836b9640 1 -- 172.21.15.1:0/1959351431 wait complete. 2024-06-18T02:30:04.027 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:03.401+0000 7f84836b9640 1 Processor -- start 2024-06-18T02:30:04.028 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:03.401+0000 7f84836b9640 1 -- start start 2024-06-18T02:30:04.028 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:03.401+0000 7f84836b9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f847c075be0 0x7f847c102280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:04.028 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:03.401+0000 7f84836b9640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f847c1027c0 con 0x7f847c075be0 2024-06-18T02:30:04.028 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:03.401+0000 7f8481455640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f847c075be0 0x7f847c102280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:30:04.028 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:03.401+0000 7f8481455640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f847c075be0 0x7f847c102280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38778/0 (socket says 172.21.15.1:38778) 2024-06-18T02:30:04.028 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:03.401+0000 7f8481455640 1 -- 172.21.15.1:0/3775278820 learned_addr learned my addr 172.21.15.1:0/3775278820 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:30:04.028 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:03.401+0000 7f8481455640 1 -- 172.21.15.1:0/3775278820 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8478009450 con 0x7f847c075be0 2024-06-18T02:30:04.029 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:03.401+0000 7f8481455640 1 --2- 172.21.15.1:0/3775278820 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f847c075be0 0x7f847c102280 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f8478031b90 tx=0x7f8478008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:30:04.029 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:03.401+0000 7f8471ffb640 1 -- 172.21.15.1:0/3775278820 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f84780328b0 con 0x7f847c075be0 2024-06-18T02:30:04.029 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:03.402+0000 7f8471ffb640 1 -- 172.21.15.1:0/3775278820 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f847803d8e0 con 0x7f847c075be0 2024-06-18T02:30:04.029 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:03.402+0000 7f84836b9640 1 -- 172.21.15.1:0/3775278820 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f847c100850 con 0x7f847c075be0 2024-06-18T02:30:04.029 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:03.402+0000 7f8471ffb640 1 -- 172.21.15.1:0/3775278820 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f847803ba20 con 0x7f847c075be0 2024-06-18T02:30:04.030 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:03.402+0000 7f84836b9640 1 -- 172.21.15.1:0/3775278820 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f847c100cc0 con 0x7f847c075be0 2024-06-18T02:30:04.030 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:03.402+0000 7f8471ffb640 1 -- 172.21.15.1:0/3775278820 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f8478039070 con 0x7f847c075be0 2024-06-18T02:30:04.030 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:03.402+0000 7f8471ffb640 1 -- 172.21.15.1:0/3775278820 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f84780468e0 con 0x7f847c075be0 2024-06-18T02:30:04.030 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:03.403+0000 7f84836b9640 1 -- 172.21.15.1:0/3775278820 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f847c04b480 con 0x7f847c075be0 2024-06-18T02:30:04.030 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:03.405+0000 7f8471ffb640 1 -- 172.21.15.1:0/3775278820 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71547 (secure 0 0 0) 0x7f8478065790 con 0x7f847c075be0 2024-06-18T02:30:04.030 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:03.466+0000 7f84836b9640 1 -- 172.21.15.1:0/3775278820 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f847c105cf0 con 0x7f847c075be0 2024-06-18T02:30:04.031 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:03.469+0000 7f8471ffb640 1 -- 172.21.15.1:0/3775278820 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f847c105cf0 con 0x7f847c075be0 2024-06-18T02:30:04.031 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:03.469+0000 7f84836b9640 1 -- 172.21.15.1:0/3775278820 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f847c075be0 msgr2=0x7f847c102280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:30:04.031 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:03.469+0000 7f84836b9640 1 --2- 172.21.15.1:0/3775278820 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f847c075be0 0x7f847c102280 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f8478031b90 tx=0x7f8478008040 comp rx=0 tx=0).stop 2024-06-18T02:30:04.031 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:03.469+0000 7f84836b9640 1 -- 172.21.15.1:0/3775278820 shutdown_connections 2024-06-18T02:30:04.031 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:03.469+0000 7f84836b9640 1 --2- 172.21.15.1:0/3775278820 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f847c075be0 0x7f847c102280 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:04.031 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:03.469+0000 7f84836b9640 1 -- 172.21.15.1:0/3775278820 >> 172.21.15.1:0/3775278820 conn(0x7f847c0fc480 msgr2=0x7f847c0fcfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:30:04.031 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:03.470+0000 7f84836b9640 1 -- 172.21.15.1:0/3775278820 shutdown_connections 2024-06-18T02:30:04.032 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:03.470+0000 7f84836b9640 1 -- 172.21.15.1:0/3775278820 wait complete. 2024-06-18T02:30:04.032 INFO:teuthology.orchestra.run.smithi001.stdout:Wrote config to /etc/ceph/ceph.conf 2024-06-18T02:30:04.032 INFO:teuthology.orchestra.run.smithi001.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-06-18T02:30:04.032 INFO:teuthology.orchestra.run.smithi001.stdout:Creating mgr... 2024-06-18T02:30:04.032 INFO:teuthology.orchestra.run.smithi001.stdout:Verifying port 9283 ... 2024-06-18T02:30:04.203 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968@mgr.smithi001.teoawz 2024-06-18T02:30:04.203 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stderr Failed to reset failed state of unit ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968@mgr.smithi001.teoawz.service: Unit ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968@mgr.smithi001.teoawz.service not loaded. 2024-06-18T02:30:04.366 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968.target.wants/ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968@mgr.smithi001.teoawz.service → /etc/systemd/system/ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968@.service. 2024-06-18T02:30:04.691 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:04 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/3775278820' entity='client.admin' 2024-06-18T02:30:04.897 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-06-18T02:30:04.897 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stdout disabled 2024-06-18T02:30:04.906 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-06-18T02:30:04.906 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stdout inactive 2024-06-18T02:30:04.906 INFO:teuthology.orchestra.run.smithi001.stdout:firewalld.service is not enabled 2024-06-18T02:30:04.906 INFO:teuthology.orchestra.run.smithi001.stdout:Not possible to enable service . firewalld.service is not available 2024-06-18T02:30:04.915 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-06-18T02:30:04.915 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stdout disabled 2024-06-18T02:30:04.923 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-06-18T02:30:04.923 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stdout inactive 2024-06-18T02:30:04.923 INFO:teuthology.orchestra.run.smithi001.stdout:firewalld.service is not enabled 2024-06-18T02:30:04.924 INFO:teuthology.orchestra.run.smithi001.stdout:Not possible to open ports <[9283]>. firewalld.service is not available 2024-06-18T02:30:04.924 INFO:teuthology.orchestra.run.smithi001.stdout:Waiting for mgr to start... 2024-06-18T02:30:04.924 INFO:teuthology.orchestra.run.smithi001.stdout:Waiting for mgr... 2024-06-18T02:30:05.627 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:05 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/1481651082' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-06-18T02:30:05.991 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 2024-06-18T02:30:05.991 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout { 2024-06-18T02:30:05.991 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "fsid": "7eefb63e-2d1a-11ef-bc9d-c7b262605968", 2024-06-18T02:30:05.991 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "health": { 2024-06-18T02:30:05.991 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-06-18T02:30:05.991 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-06-18T02:30:05.991 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-06-18T02:30:05.991 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-06-18T02:30:05.991 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-06-18T02:30:05.992 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-06-18T02:30:05.992 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 0 2024-06-18T02:30:05.992 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout ], 2024-06-18T02:30:05.992 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-06-18T02:30:05.992 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "smithi001" 2024-06-18T02:30:05.992 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout ], 2024-06-18T02:30:05.992 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-06-18T02:30:05.992 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "monmap": { 2024-06-18T02:30:05.992 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-18T02:30:05.992 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-06-18T02:30:05.992 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-06-18T02:30:05.992 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-06-18T02:30:05.992 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-06-18T02:30:05.992 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-18T02:30:05.993 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-06-18T02:30:05.993 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-06-18T02:30:05.993 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-06-18T02:30:05.993 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-06-18T02:30:05.993 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-06-18T02:30:05.993 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-06-18T02:30:05.993 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-06-18T02:30:05.993 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-06-18T02:30:05.993 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-06-18T02:30:05.993 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-06-18T02:30:05.993 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-06-18T02:30:05.993 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-06-18T02:30:05.993 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-06-18T02:30:05.993 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-06-18T02:30:05.994 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-06-18T02:30:05.994 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-06-18T02:30:05.994 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-06-18T02:30:05.994 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-06-18T02:30:05.994 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-18T02:30:05.994 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-06-18T02:30:05.994 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-06-18T02:30:05.994 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-06-18T02:30:05.994 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-06-18T02:30:05.994 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "available": false, 2024-06-18T02:30:05.994 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-06-18T02:30:05.994 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "modules": [ 2024-06-18T02:30:05.994 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "iostat", 2024-06-18T02:30:05.994 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "nfs", 2024-06-18T02:30:05.995 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "restful" 2024-06-18T02:30:05.995 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout ], 2024-06-18T02:30:05.995 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-18T02:30:05.995 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-06-18T02:30:05.995 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-06-18T02:30:05.995 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-18T02:30:05.995 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "modified": "2024-06-18T02:29:58.467445+0000", 2024-06-18T02:30:05.995 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-18T02:30:05.995 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-06-18T02:30:05.995 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-06-18T02:30:05.995 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout } 2024-06-18T02:30:05.995 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:05.305+0000 7fa937234640 1 Processor -- start 2024-06-18T02:30:05.995 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:05.305+0000 7fa937234640 1 -- start start 2024-06-18T02:30:05.996 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:05.305+0000 7fa937234640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa930075b80 0x7fa930106d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:05.996 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:05.305+0000 7fa937234640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa930075f90 con 0x7fa930075b80 2024-06-18T02:30:05.996 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:05.305+0000 7fa934fd0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa930075b80 0x7fa930106d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:30:05.996 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:05.305+0000 7fa934fd0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa930075b80 0x7fa930106d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53920/0 (socket says 172.21.15.1:53920) 2024-06-18T02:30:05.996 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:05.305+0000 7fa934fd0640 1 -- 172.21.15.1:0/1347981571 learned_addr learned my addr 172.21.15.1:0/1347981571 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:30:05.996 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:05.306+0000 7fa934fd0640 1 -- 172.21.15.1:0/1347981571 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa930102770 con 0x7fa930075b80 2024-06-18T02:30:05.996 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:05.306+0000 7fa934fd0640 1 --2- 172.21.15.1:0/1347981571 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa930075b80 0x7fa930106d80 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa924009840 tx=0x7fa9240091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35115e0dba2713c8 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:30:05.996 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:05.306+0000 7fa92f7fe640 1 -- 172.21.15.1:0/1347981571 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa924032440 con 0x7fa930075b80 2024-06-18T02:30:05.996 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:05.306+0000 7fa92f7fe640 1 -- 172.21.15.1:0/1347981571 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fa924032a30 con 0x7fa930075b80 2024-06-18T02:30:05.996 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:05.306+0000 7fa92f7fe640 1 -- 172.21.15.1:0/1347981571 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa924031ad0 con 0x7fa930075b80 2024-06-18T02:30:05.996 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:05.307+0000 7fa937234640 1 -- 172.21.15.1:0/1347981571 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa930075b80 msgr2=0x7fa930106d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:30:05.996 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:05.307+0000 7fa937234640 1 --2- 172.21.15.1:0/1347981571 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa930075b80 0x7fa930106d80 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa924009840 tx=0x7fa9240091d0 comp rx=0 tx=0).stop 2024-06-18T02:30:05.996 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:05.307+0000 7fa937234640 1 -- 172.21.15.1:0/1347981571 shutdown_connections 2024-06-18T02:30:05.997 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:05.307+0000 7fa937234640 1 --2- 172.21.15.1:0/1347981571 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa930075b80 0x7fa930106d80 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:05.997 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:05.307+0000 7fa937234640 1 -- 172.21.15.1:0/1347981571 >> 172.21.15.1:0/1347981571 conn(0x7fa9300fc450 msgr2=0x7fa9300fe880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:30:05.997 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:05.307+0000 7fa937234640 1 -- 172.21.15.1:0/1347981571 shutdown_connections 2024-06-18T02:30:05.997 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:05.307+0000 7fa937234640 1 -- 172.21.15.1:0/1347981571 wait complete. 2024-06-18T02:30:05.997 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:05.308+0000 7fa937234640 1 Processor -- start 2024-06-18T02:30:05.997 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:05.308+0000 7fa937234640 1 -- start start 2024-06-18T02:30:05.997 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:05.308+0000 7fa937234640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa930075b80 0x7fa930112c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:05.997 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:05.308+0000 7fa937234640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9301131c0 con 0x7fa930075b80 2024-06-18T02:30:05.997 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:05.308+0000 7fa934fd0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa930075b80 0x7fa930112c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:30:05.997 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:05.308+0000 7fa934fd0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa930075b80 0x7fa930112c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53922/0 (socket says 172.21.15.1:53922) 2024-06-18T02:30:05.997 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:05.308+0000 7fa934fd0640 1 -- 172.21.15.1:0/1481651082 learned_addr learned my addr 172.21.15.1:0/1481651082 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:30:05.997 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:05.309+0000 7fa934fd0640 1 -- 172.21.15.1:0/1481651082 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa924009450 con 0x7fa930075b80 2024-06-18T02:30:05.997 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:05.309+0000 7fa934fd0640 1 --2- 172.21.15.1:0/1481651082 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa930075b80 0x7fa930112c80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa924006f00 tx=0x7fa924008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:30:05.997 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:05.309+0000 7fa92d7fa640 1 -- 172.21.15.1:0/1481651082 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa92403d450 con 0x7fa930075b80 2024-06-18T02:30:05.998 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:05.309+0000 7fa937234640 1 -- 172.21.15.1:0/1481651082 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa930113420 con 0x7fa930075b80 2024-06-18T02:30:05.998 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:05.309+0000 7fa92d7fa640 1 -- 172.21.15.1:0/1481651082 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fa92403da40 con 0x7fa930075b80 2024-06-18T02:30:05.998 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:05.309+0000 7fa92d7fa640 1 -- 172.21.15.1:0/1481651082 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa92403bb10 con 0x7fa930075b80 2024-06-18T02:30:05.998 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:05.309+0000 7fa937234640 1 -- 172.21.15.1:0/1481651082 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa930113f50 con 0x7fa930075b80 2024-06-18T02:30:05.998 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:05.310+0000 7fa92d7fa640 1 -- 172.21.15.1:0/1481651082 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7fa924039070 con 0x7fa930075b80 2024-06-18T02:30:05.998 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:05.310+0000 7fa92d7fa640 1 -- 172.21.15.1:0/1481651082 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fa9240468e0 con 0x7fa930075b80 2024-06-18T02:30:05.998 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:05.311+0000 7fa937234640 1 -- 172.21.15.1:0/1481651082 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa93005ead0 con 0x7fa930075b80 2024-06-18T02:30:05.998 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:05.313+0000 7fa92d7fa640 1 -- 172.21.15.1:0/1481651082 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71547 (secure 0 0 0) 0x7fa924041020 con 0x7fa930075b80 2024-06-18T02:30:05.998 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:05.377+0000 7fa937234640 1 -- 172.21.15.1:0/1481651082 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fa930107ee0 con 0x7fa930075b80 2024-06-18T02:30:05.998 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:05.377+0000 7fa92d7fa640 1 -- 172.21.15.1:0/1481651082 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7fa924037070 con 0x7fa930075b80 2024-06-18T02:30:05.998 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:05.378+0000 7fa937234640 1 -- 172.21.15.1:0/1481651082 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa930075b80 msgr2=0x7fa930112c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:30:05.998 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:05.378+0000 7fa937234640 1 --2- 172.21.15.1:0/1481651082 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa930075b80 0x7fa930112c80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa924006f00 tx=0x7fa924008040 comp rx=0 tx=0).stop 2024-06-18T02:30:05.998 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:05.378+0000 7fa937234640 1 -- 172.21.15.1:0/1481651082 shutdown_connections 2024-06-18T02:30:05.999 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:05.378+0000 7fa937234640 1 --2- 172.21.15.1:0/1481651082 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa930075b80 0x7fa930112c80 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:05.999 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:05.378+0000 7fa937234640 1 -- 172.21.15.1:0/1481651082 >> 172.21.15.1:0/1481651082 conn(0x7fa9300fc450 msgr2=0x7fa9300fd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:30:05.999 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:05.378+0000 7fa937234640 1 -- 172.21.15.1:0/1481651082 shutdown_connections 2024-06-18T02:30:05.999 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:05.378+0000 7fa937234640 1 -- 172.21.15.1:0/1481651082 wait complete. 2024-06-18T02:30:05.999 INFO:teuthology.orchestra.run.smithi001.stdout:mgr not available, waiting (1/15)... 2024-06-18T02:30:08.657 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:08 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/1150365845' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-06-18T02:30:09.116 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 2024-06-18T02:30:09.116 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout { 2024-06-18T02:30:09.116 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "fsid": "7eefb63e-2d1a-11ef-bc9d-c7b262605968", 2024-06-18T02:30:09.116 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "health": { 2024-06-18T02:30:09.116 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-06-18T02:30:09.116 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-06-18T02:30:09.117 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-06-18T02:30:09.117 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-06-18T02:30:09.117 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-06-18T02:30:09.117 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-06-18T02:30:09.117 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 0 2024-06-18T02:30:09.117 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout ], 2024-06-18T02:30:09.117 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-06-18T02:30:09.117 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "smithi001" 2024-06-18T02:30:09.117 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout ], 2024-06-18T02:30:09.117 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-06-18T02:30:09.117 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "monmap": { 2024-06-18T02:30:09.117 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-18T02:30:09.117 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-06-18T02:30:09.117 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-06-18T02:30:09.118 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-06-18T02:30:09.118 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-06-18T02:30:09.118 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-18T02:30:09.118 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-06-18T02:30:09.118 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-06-18T02:30:09.118 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-06-18T02:30:09.118 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-06-18T02:30:09.118 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-06-18T02:30:09.125 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-06-18T02:30:09.125 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-06-18T02:30:09.125 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-06-18T02:30:09.125 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-06-18T02:30:09.125 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-06-18T02:30:09.125 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-06-18T02:30:09.126 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-06-18T02:30:09.126 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-06-18T02:30:09.126 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-06-18T02:30:09.126 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-06-18T02:30:09.126 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-06-18T02:30:09.126 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-06-18T02:30:09.126 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-06-18T02:30:09.126 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-18T02:30:09.126 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-06-18T02:30:09.126 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-06-18T02:30:09.126 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-06-18T02:30:09.126 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-06-18T02:30:09.126 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "available": false, 2024-06-18T02:30:09.127 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-06-18T02:30:09.127 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "modules": [ 2024-06-18T02:30:09.127 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "iostat", 2024-06-18T02:30:09.127 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "nfs", 2024-06-18T02:30:09.127 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "restful" 2024-06-18T02:30:09.127 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout ], 2024-06-18T02:30:09.127 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-18T02:30:09.127 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-06-18T02:30:09.127 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-06-18T02:30:09.127 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-18T02:30:09.127 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "modified": "2024-06-18T02:29:58.467445+0000", 2024-06-18T02:30:09.127 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-18T02:30:09.127 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-06-18T02:30:09.127 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-06-18T02:30:09.127 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout } 2024-06-18T02:30:09.128 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:08.409+0000 7fc1f1d3f640 1 Processor -- start 2024-06-18T02:30:09.128 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:08.409+0000 7fc1f1d3f640 1 -- start start 2024-06-18T02:30:09.128 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:08.410+0000 7fc1f1d3f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1ec100c90 0x7fc1ec1010a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:09.128 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:08.410+0000 7fc1f1d3f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1ec1015e0 con 0x7fc1ec100c90 2024-06-18T02:30:09.128 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:08.410+0000 7fc1eb7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1ec100c90 0x7fc1ec1010a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:30:09.128 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:08.410+0000 7fc1eb7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1ec100c90 0x7fc1ec1010a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53932/0 (socket says 172.21.15.1:53932) 2024-06-18T02:30:09.128 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:08.410+0000 7fc1eb7fe640 1 -- 172.21.15.1:0/1321369827 learned_addr learned my addr 172.21.15.1:0/1321369827 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:30:09.128 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:08.410+0000 7fc1eb7fe640 1 -- 172.21.15.1:0/1321369827 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1ec101720 con 0x7fc1ec100c90 2024-06-18T02:30:09.128 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:08.411+0000 7fc1eb7fe640 1 --2- 172.21.15.1:0/1321369827 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1ec100c90 0x7fc1ec1010a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc1e0009840 tx=0x7fc1e00091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=141040879db853e6 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:30:09.128 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:08.411+0000 7fc1ea7fc640 1 -- 172.21.15.1:0/1321369827 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc1e0032440 con 0x7fc1ec100c90 2024-06-18T02:30:09.128 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:08.411+0000 7fc1ea7fc640 1 -- 172.21.15.1:0/1321369827 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fc1e0032a30 con 0x7fc1ec100c90 2024-06-18T02:30:09.128 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:08.411+0000 7fc1ea7fc640 1 -- 172.21.15.1:0/1321369827 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc1e0031ad0 con 0x7fc1ec100c90 2024-06-18T02:30:09.129 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:08.412+0000 7fc1f1d3f640 1 -- 172.21.15.1:0/1321369827 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1ec100c90 msgr2=0x7fc1ec1010a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:30:09.129 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:08.412+0000 7fc1f1d3f640 1 --2- 172.21.15.1:0/1321369827 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1ec100c90 0x7fc1ec1010a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc1e0009840 tx=0x7fc1e00091d0 comp rx=0 tx=0).stop 2024-06-18T02:30:09.129 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:08.412+0000 7fc1f1d3f640 1 -- 172.21.15.1:0/1321369827 shutdown_connections 2024-06-18T02:30:09.129 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:08.412+0000 7fc1f1d3f640 1 --2- 172.21.15.1:0/1321369827 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1ec100c90 0x7fc1ec1010a0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:09.129 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:08.412+0000 7fc1f1d3f640 1 -- 172.21.15.1:0/1321369827 >> 172.21.15.1:0/1321369827 conn(0x7fc1ec0fc450 msgr2=0x7fc1ec0fe880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:30:09.129 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:08.412+0000 7fc1f1d3f640 1 -- 172.21.15.1:0/1321369827 shutdown_connections 2024-06-18T02:30:09.129 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:08.412+0000 7fc1f1d3f640 1 -- 172.21.15.1:0/1321369827 wait complete. 2024-06-18T02:30:09.129 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:08.412+0000 7fc1f1d3f640 1 Processor -- start 2024-06-18T02:30:09.129 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:08.412+0000 7fc1f1d3f640 1 -- start start 2024-06-18T02:30:09.129 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:08.413+0000 7fc1f1d3f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1ec100c90 0x7fc1ec075b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:09.129 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:08.413+0000 7fc1f1d3f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1ec113810 con 0x7fc1ec100c90 2024-06-18T02:30:09.129 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:08.413+0000 7fc1eb7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1ec100c90 0x7fc1ec075b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:30:09.129 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:08.413+0000 7fc1eb7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1ec100c90 0x7fc1ec075b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53936/0 (socket says 172.21.15.1:53936) 2024-06-18T02:30:09.129 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:08.413+0000 7fc1eb7fe640 1 -- 172.21.15.1:0/1150365845 learned_addr learned my addr 172.21.15.1:0/1150365845 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:30:09.129 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:08.413+0000 7fc1eb7fe640 1 -- 172.21.15.1:0/1150365845 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1e0009450 con 0x7fc1ec100c90 2024-06-18T02:30:09.130 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:08.413+0000 7fc1eb7fe640 1 --2- 172.21.15.1:0/1150365845 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1ec100c90 0x7fc1ec075b40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc1e0009420 tx=0x7fc1e0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:30:09.130 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:08.413+0000 7fc1d3fff640 1 -- 172.21.15.1:0/1150365845 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc1e003d5b0 con 0x7fc1ec100c90 2024-06-18T02:30:09.130 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:08.414+0000 7fc1f1d3f640 1 -- 172.21.15.1:0/1150365845 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1ec113950 con 0x7fc1ec100c90 2024-06-18T02:30:09.130 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:08.414+0000 7fc1d3fff640 1 -- 172.21.15.1:0/1150365845 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fc1e003dba0 con 0x7fc1ec100c90 2024-06-18T02:30:09.130 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:08.414+0000 7fc1d3fff640 1 -- 172.21.15.1:0/1150365845 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc1e003bb40 con 0x7fc1ec100c90 2024-06-18T02:30:09.130 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:08.414+0000 7fc1f1d3f640 1 -- 172.21.15.1:0/1150365845 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1ec113dc0 con 0x7fc1ec100c90 2024-06-18T02:30:09.130 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:08.414+0000 7fc1d3fff640 1 -- 172.21.15.1:0/1150365845 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7fc1e0039070 con 0x7fc1ec100c90 2024-06-18T02:30:09.130 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:08.414+0000 7fc1d3fff640 1 -- 172.21.15.1:0/1150365845 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fc1e0046950 con 0x7fc1ec100c90 2024-06-18T02:30:09.130 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:08.415+0000 7fc1f1d3f640 1 -- 172.21.15.1:0/1150365845 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1ec04b480 con 0x7fc1ec100c90 2024-06-18T02:30:09.130 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:08.417+0000 7fc1d3fff640 1 -- 172.21.15.1:0/1150365845 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71547 (secure 0 0 0) 0x7fc1e0041020 con 0x7fc1ec100c90 2024-06-18T02:30:09.130 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:08.481+0000 7fc1f1d3f640 1 -- 172.21.15.1:0/1150365845 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fc1ec1029b0 con 0x7fc1ec100c90 2024-06-18T02:30:09.130 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:08.481+0000 7fc1d3fff640 1 -- 172.21.15.1:0/1150365845 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7fc1e0037070 con 0x7fc1ec100c90 2024-06-18T02:30:09.130 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:08.482+0000 7fc1f1d3f640 1 -- 172.21.15.1:0/1150365845 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1ec100c90 msgr2=0x7fc1ec075b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:30:09.131 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:08.482+0000 7fc1f1d3f640 1 --2- 172.21.15.1:0/1150365845 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1ec100c90 0x7fc1ec075b40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc1e0009420 tx=0x7fc1e0008040 comp rx=0 tx=0).stop 2024-06-18T02:30:09.131 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:08.482+0000 7fc1f1d3f640 1 -- 172.21.15.1:0/1150365845 shutdown_connections 2024-06-18T02:30:09.131 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:08.482+0000 7fc1f1d3f640 1 --2- 172.21.15.1:0/1150365845 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1ec100c90 0x7fc1ec075b40 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:09.131 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:08.482+0000 7fc1f1d3f640 1 -- 172.21.15.1:0/1150365845 >> 172.21.15.1:0/1150365845 conn(0x7fc1ec0fc450 msgr2=0x7fc1ec0fd040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:30:09.131 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:08.482+0000 7fc1f1d3f640 1 -- 172.21.15.1:0/1150365845 shutdown_connections 2024-06-18T02:30:09.131 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:08.482+0000 7fc1f1d3f640 1 -- 172.21.15.1:0/1150365845 wait complete. 2024-06-18T02:30:09.131 INFO:teuthology.orchestra.run.smithi001.stdout:mgr not available, waiting (2/15)... 2024-06-18T02:30:09.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:09 smithi001 ceph-mon[29099]: Activating manager daemon smithi001.teoawz 2024-06-18T02:30:09.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:09 smithi001 ceph-mon[29099]: mgrmap e2: smithi001.teoawz(active, starting, since 0.00279841s) 2024-06-18T02:30:09.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:09 smithi001 ceph-mon[29099]: from='mgr.14100 172.21.15.1:0/1166871228' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-18T02:30:09.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:09 smithi001 ceph-mon[29099]: from='mgr.14100 172.21.15.1:0/1166871228' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-18T02:30:09.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:09 smithi001 ceph-mon[29099]: from='mgr.14100 172.21.15.1:0/1166871228' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-18T02:30:09.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:09 smithi001 ceph-mon[29099]: from='mgr.14100 172.21.15.1:0/1166871228' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "mon metadata", "id": "smithi001"}]: dispatch 2024-06-18T02:30:09.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:09 smithi001 ceph-mon[29099]: from='mgr.14100 172.21.15.1:0/1166871228' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "mgr metadata", "who": "smithi001.teoawz", "id": "smithi001.teoawz"}]: dispatch 2024-06-18T02:30:09.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:09 smithi001 ceph-mon[29099]: Manager daemon smithi001.teoawz is now available 2024-06-18T02:30:09.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:09 smithi001 ceph-mon[29099]: from='mgr.14100 172.21.15.1:0/1166871228' entity='mgr.smithi001.teoawz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi001.teoawz/mirror_snapshot_schedule"}]: dispatch 2024-06-18T02:30:09.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:09 smithi001 ceph-mon[29099]: from='mgr.14100 172.21.15.1:0/1166871228' entity='mgr.smithi001.teoawz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi001.teoawz/trash_purge_schedule"}]: dispatch 2024-06-18T02:30:09.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:09 smithi001 ceph-mon[29099]: from='mgr.14100 172.21.15.1:0/1166871228' entity='mgr.smithi001.teoawz' 2024-06-18T02:30:09.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:09 smithi001 ceph-mon[29099]: from='mgr.14100 172.21.15.1:0/1166871228' entity='mgr.smithi001.teoawz' 2024-06-18T02:30:09.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:09 smithi001 ceph-mon[29099]: from='mgr.14100 172.21.15.1:0/1166871228' entity='mgr.smithi001.teoawz' 2024-06-18T02:30:11.773 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:11 smithi001 ceph-mon[29099]: mgrmap e3: smithi001.teoawz(active, since 1.0065s) 2024-06-18T02:30:12.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:12 smithi001 ceph-mon[29099]: mgrmap e4: smithi001.teoawz(active, since 2s) 2024-06-18T02:30:12.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:12 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/3458013177' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-06-18T02:30:17.185 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 2024-06-18T02:30:17.186 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout { 2024-06-18T02:30:17.186 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "fsid": "7eefb63e-2d1a-11ef-bc9d-c7b262605968", 2024-06-18T02:30:17.186 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "health": { 2024-06-18T02:30:17.186 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-06-18T02:30:17.186 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-06-18T02:30:17.186 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-06-18T02:30:17.186 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-06-18T02:30:17.186 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-06-18T02:30:17.186 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-06-18T02:30:17.186 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 0 2024-06-18T02:30:17.186 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout ], 2024-06-18T02:30:17.186 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-06-18T02:30:17.187 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "smithi001" 2024-06-18T02:30:17.187 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout ], 2024-06-18T02:30:17.187 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-06-18T02:30:17.187 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "monmap": { 2024-06-18T02:30:17.187 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-18T02:30:17.187 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-06-18T02:30:17.187 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-06-18T02:30:17.187 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-06-18T02:30:17.187 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-06-18T02:30:17.187 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-18T02:30:17.187 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-06-18T02:30:17.187 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-06-18T02:30:17.187 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-06-18T02:30:17.196 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-06-18T02:30:17.196 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-06-18T02:30:17.196 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-06-18T02:30:17.196 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-06-18T02:30:17.196 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-06-18T02:30:17.197 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-06-18T02:30:17.197 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-06-18T02:30:17.197 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-06-18T02:30:17.197 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-06-18T02:30:17.197 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-06-18T02:30:17.197 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-06-18T02:30:17.197 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-06-18T02:30:17.197 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-06-18T02:30:17.197 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-06-18T02:30:17.197 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-06-18T02:30:17.197 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-18T02:30:17.197 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-06-18T02:30:17.197 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-06-18T02:30:17.198 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-06-18T02:30:17.198 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-06-18T02:30:17.198 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "available": true, 2024-06-18T02:30:17.198 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-06-18T02:30:17.198 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "modules": [ 2024-06-18T02:30:17.198 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "iostat", 2024-06-18T02:30:17.198 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "nfs", 2024-06-18T02:30:17.198 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "restful" 2024-06-18T02:30:17.198 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout ], 2024-06-18T02:30:17.198 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-18T02:30:17.198 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-06-18T02:30:17.198 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-06-18T02:30:17.199 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-18T02:30:17.199 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "modified": "2024-06-18T02:29:58.467445+0000", 2024-06-18T02:30:17.199 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-18T02:30:17.199 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout }, 2024-06-18T02:30:17.199 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-06-18T02:30:17.199 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout } 2024-06-18T02:30:17.199 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:11.523+0000 7fce59872640 1 Processor -- start 2024-06-18T02:30:17.199 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:11.523+0000 7fce59872640 1 -- start start 2024-06-18T02:30:17.199 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:11.523+0000 7fce59872640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce54100c90 0x7fce541010a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:17.199 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:11.523+0000 7fce59872640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce541015e0 con 0x7fce54100c90 2024-06-18T02:30:17.199 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:11.524+0000 7fce52ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce54100c90 0x7fce541010a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:30:17.200 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:11.524+0000 7fce52ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce54100c90 0x7fce541010a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53998/0 (socket says 172.21.15.1:53998) 2024-06-18T02:30:17.200 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:11.524+0000 7fce52ffd640 1 -- 172.21.15.1:0/2960145162 learned_addr learned my addr 172.21.15.1:0/2960145162 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:30:17.200 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:11.524+0000 7fce52ffd640 1 -- 172.21.15.1:0/2960145162 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce54101720 con 0x7fce54100c90 2024-06-18T02:30:17.200 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:11.525+0000 7fce52ffd640 1 --2- 172.21.15.1:0/2960145162 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce54100c90 0x7fce541010a0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fce44009e50 tx=0x7fce44009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=711b3e3e22bb742f server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:30:17.200 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:11.525+0000 7fce51ffb640 1 -- 172.21.15.1:0/2960145162 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce44031560 con 0x7fce54100c90 2024-06-18T02:30:17.200 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:11.525+0000 7fce51ffb640 1 -- 172.21.15.1:0/2960145162 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fce4402d050 con 0x7fce54100c90 2024-06-18T02:30:17.200 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:11.525+0000 7fce51ffb640 1 -- 172.21.15.1:0/2960145162 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce44007a50 con 0x7fce54100c90 2024-06-18T02:30:17.200 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:11.526+0000 7fce59872640 1 -- 172.21.15.1:0/2960145162 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce54100c90 msgr2=0x7fce541010a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:30:17.200 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:11.526+0000 7fce59872640 1 --2- 172.21.15.1:0/2960145162 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce54100c90 0x7fce541010a0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fce44009e50 tx=0x7fce44009840 comp rx=0 tx=0).stop 2024-06-18T02:30:17.200 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:11.526+0000 7fce59872640 1 -- 172.21.15.1:0/2960145162 shutdown_connections 2024-06-18T02:30:17.200 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:11.526+0000 7fce59872640 1 --2- 172.21.15.1:0/2960145162 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce54100c90 0x7fce541010a0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:17.200 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:11.526+0000 7fce59872640 1 -- 172.21.15.1:0/2960145162 >> 172.21.15.1:0/2960145162 conn(0x7fce540fc450 msgr2=0x7fce540fe880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:30:17.201 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:11.526+0000 7fce59872640 1 -- 172.21.15.1:0/2960145162 shutdown_connections 2024-06-18T02:30:17.201 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:11.526+0000 7fce59872640 1 -- 172.21.15.1:0/2960145162 wait complete. 2024-06-18T02:30:17.201 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:11.526+0000 7fce59872640 1 Processor -- start 2024-06-18T02:30:17.201 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:11.526+0000 7fce59872640 1 -- start start 2024-06-18T02:30:17.201 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:11.527+0000 7fce59872640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce54100c90 0x7fce54106e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:17.201 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:11.527+0000 7fce59872640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce541073a0 con 0x7fce54100c90 2024-06-18T02:30:17.201 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:11.527+0000 7fce52ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce54100c90 0x7fce54106e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:30:17.201 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:11.527+0000 7fce52ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce54100c90 0x7fce54106e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54006/0 (socket says 172.21.15.1:54006) 2024-06-18T02:30:17.201 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:11.527+0000 7fce52ffd640 1 -- 172.21.15.1:0/3458013177 learned_addr learned my addr 172.21.15.1:0/3458013177 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:30:17.201 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:11.527+0000 7fce52ffd640 1 -- 172.21.15.1:0/3458013177 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce44009ae0 con 0x7fce54100c90 2024-06-18T02:30:17.201 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:11.527+0000 7fce52ffd640 1 --2- 172.21.15.1:0/3458013177 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce54100c90 0x7fce54106e60 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fce44007730 tx=0x7fce44009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:30:17.201 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:11.527+0000 7fce4b7fe640 1 -- 172.21.15.1:0/3458013177 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce44031de0 con 0x7fce54100c90 2024-06-18T02:30:17.202 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:11.528+0000 7fce4b7fe640 1 -- 172.21.15.1:0/3458013177 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fce44007e10 con 0x7fce54100c90 2024-06-18T02:30:17.202 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:11.528+0000 7fce59872640 1 -- 172.21.15.1:0/3458013177 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce54105430 con 0x7fce54100c90 2024-06-18T02:30:17.202 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:11.528+0000 7fce4b7fe640 1 -- 172.21.15.1:0/3458013177 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce4403a5a0 con 0x7fce54100c90 2024-06-18T02:30:17.202 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:11.528+0000 7fce59872640 1 -- 172.21.15.1:0/3458013177 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce541058a0 con 0x7fce54100c90 2024-06-18T02:30:17.202 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:11.528+0000 7fce4b7fe640 1 -- 172.21.15.1:0/3458013177 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 3) v1 ==== 41829+0+0 (secure 0 0 0) 0x7fce44038040 con 0x7fce54100c90 2024-06-18T02:30:17.202 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:11.529+0000 7fce59872640 1 -- 172.21.15.1:0/3458013177 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce5404b480 con 0x7fce54100c90 2024-06-18T02:30:17.202 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:11.529+0000 7fce4b7fe640 1 --2- 172.21.15.1:0/3458013177 >> [v2:172.21.15.1:6800/752464819,v1:172.21.15.1:6801/752464819] conn(0x7fce400346e0 0x7fce40036b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:17.202 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:11.529+0000 7fce4b7fe640 1 -- 172.21.15.1:0/3458013177 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fce44040030 con 0x7fce54100c90 2024-06-18T02:30:17.202 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:11.529+0000 7fce527fc640 1 --2- 172.21.15.1:0/3458013177 >> [v2:172.21.15.1:6800/752464819,v1:172.21.15.1:6801/752464819] conn(0x7fce400346e0 0x7fce40036b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:30:17.202 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:11.529+0000 7fce527fc640 1 --2- 172.21.15.1:0/3458013177 >> [v2:172.21.15.1:6800/752464819,v1:172.21.15.1:6801/752464819] conn(0x7fce400346e0 0x7fce40036b90 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fce3c006fd0 tx=0x7fce3c006db0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:30:17.202 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:11.534+0000 7fce4b7fe640 1 -- 172.21.15.1:0/3458013177 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fce4404c950 con 0x7fce54100c90 2024-06-18T02:30:17.203 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:11.544+0000 7fce4b7fe640 1 -- 172.21.15.1:0/3458013177 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 4) v1 ==== 41935+0+0 (secure 0 0 0) 0x7fce44045a50 con 0x7fce54100c90 2024-06-18T02:30:17.203 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:11.773+0000 7fce59872640 1 -- 172.21.15.1:0/3458013177 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fce541029b0 con 0x7fce54100c90 2024-06-18T02:30:17.203 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:11.774+0000 7fce4b7fe640 1 -- 172.21.15.1:0/3458013177 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1247 (secure 0 0 0) 0x7fce440694f0 con 0x7fce54100c90 2024-06-18T02:30:17.203 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:11.776+0000 7fce59872640 1 -- 172.21.15.1:0/3458013177 >> [v2:172.21.15.1:6800/752464819,v1:172.21.15.1:6801/752464819] conn(0x7fce400346e0 msgr2=0x7fce40036b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:30:17.203 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:11.776+0000 7fce59872640 1 --2- 172.21.15.1:0/3458013177 >> [v2:172.21.15.1:6800/752464819,v1:172.21.15.1:6801/752464819] conn(0x7fce400346e0 0x7fce40036b90 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fce3c006fd0 tx=0x7fce3c006db0 comp rx=0 tx=0).stop 2024-06-18T02:30:17.203 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:11.776+0000 7fce59872640 1 -- 172.21.15.1:0/3458013177 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce54100c90 msgr2=0x7fce54106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:30:17.203 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:11.776+0000 7fce59872640 1 --2- 172.21.15.1:0/3458013177 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce54100c90 0x7fce54106e60 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fce44007730 tx=0x7fce44009840 comp rx=0 tx=0).stop 2024-06-18T02:30:17.203 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:11.776+0000 7fce59872640 1 -- 172.21.15.1:0/3458013177 shutdown_connections 2024-06-18T02:30:17.203 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:11.776+0000 7fce59872640 1 --2- 172.21.15.1:0/3458013177 >> [v2:172.21.15.1:6800/752464819,v1:172.21.15.1:6801/752464819] conn(0x7fce400346e0 0x7fce40036b90 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:17.203 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:11.776+0000 7fce59872640 1 --2- 172.21.15.1:0/3458013177 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce54100c90 0x7fce54106e60 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:17.203 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:11.776+0000 7fce59872640 1 -- 172.21.15.1:0/3458013177 >> 172.21.15.1:0/3458013177 conn(0x7fce540fc450 msgr2=0x7fce540fd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:30:17.203 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:11.776+0000 7fce59872640 1 -- 172.21.15.1:0/3458013177 shutdown_connections 2024-06-18T02:30:17.203 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:11.776+0000 7fce59872640 1 -- 172.21.15.1:0/3458013177 wait complete. 2024-06-18T02:30:17.204 INFO:teuthology.orchestra.run.smithi001.stdout:mgr is available 2024-06-18T02:30:20.377 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:20 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/1214209948' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-06-18T02:30:23.801 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 2024-06-18T02:30:23.801 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout [global] 2024-06-18T02:30:23.801 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout fsid = 7eefb63e-2d1a-11ef-bc9d-c7b262605968 2024-06-18T02:30:23.801 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-06-18T02:30:23.801 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-06-18T02:30:23.801 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-06-18T02:30:23.801 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-06-18T02:30:23.801 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 2024-06-18T02:30:23.801 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout [mgr] 2024-06-18T02:30:23.801 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-06-18T02:30:23.801 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 2024-06-18T02:30:23.801 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout [osd] 2024-06-18T02:30:23.802 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-06-18T02:30:23.802 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-06-18T02:30:23.802 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-06-18T02:30:23.802 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-06-18T02:30:23.802 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-06-18T02:30:23.802 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:20.111+0000 7f11053e2640 1 Processor -- start 2024-06-18T02:30:23.802 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:20.111+0000 7f11053e2640 1 -- start start 2024-06-18T02:30:23.802 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:20.111+0000 7f11053e2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1100101d00 0x7f1100102110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:23.802 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:20.111+0000 7f11053e2640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1100102650 con 0x7f1100101d00 2024-06-18T02:30:23.802 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:20.111+0000 7f10feffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1100101d00 0x7f1100102110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:30:23.802 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:20.111+0000 7f10feffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1100101d00 0x7f1100102110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34902/0 (socket says 172.21.15.1:34902) 2024-06-18T02:30:23.802 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:20.111+0000 7f10feffd640 1 -- 172.21.15.1:0/1041322059 learned_addr learned my addr 172.21.15.1:0/1041322059 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:30:23.803 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:20.111+0000 7f10feffd640 1 -- 172.21.15.1:0/1041322059 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1100102790 con 0x7f1100101d00 2024-06-18T02:30:23.803 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:20.112+0000 7f10feffd640 1 --2- 172.21.15.1:0/1041322059 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1100101d00 0x7f1100102110 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f10f0009840 tx=0x7f10f00091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ace3965a6fbd493 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:30:23.803 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:20.112+0000 7f10fdffb640 1 -- 172.21.15.1:0/1041322059 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f10f0032440 con 0x7f1100101d00 2024-06-18T02:30:23.803 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:20.112+0000 7f10fdffb640 1 -- 172.21.15.1:0/1041322059 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f10f0032a30 con 0x7f1100101d00 2024-06-18T02:30:23.803 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:20.112+0000 7f10fdffb640 1 -- 172.21.15.1:0/1041322059 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f10f0031ad0 con 0x7f1100101d00 2024-06-18T02:30:23.803 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:20.113+0000 7f11053e2640 1 -- 172.21.15.1:0/1041322059 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1100101d00 msgr2=0x7f1100102110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:30:23.803 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:20.113+0000 7f11053e2640 1 --2- 172.21.15.1:0/1041322059 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1100101d00 0x7f1100102110 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f10f0009840 tx=0x7f10f00091d0 comp rx=0 tx=0).stop 2024-06-18T02:30:23.803 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:20.113+0000 7f11053e2640 1 -- 172.21.15.1:0/1041322059 shutdown_connections 2024-06-18T02:30:23.803 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:20.113+0000 7f11053e2640 1 --2- 172.21.15.1:0/1041322059 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1100101d00 0x7f1100102110 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:23.803 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:20.113+0000 7f11053e2640 1 -- 172.21.15.1:0/1041322059 >> 172.21.15.1:0/1041322059 conn(0x7f11000fd500 msgr2=0x7f11000ff910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:30:23.803 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:20.113+0000 7f11053e2640 1 -- 172.21.15.1:0/1041322059 shutdown_connections 2024-06-18T02:30:23.803 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:20.113+0000 7f11053e2640 1 -- 172.21.15.1:0/1041322059 wait complete. 2024-06-18T02:30:23.803 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:20.114+0000 7f11053e2640 1 Processor -- start 2024-06-18T02:30:23.804 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:20.114+0000 7f11053e2640 1 -- start start 2024-06-18T02:30:23.804 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:20.114+0000 7f11053e2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1100101d00 0x7f1100110ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:23.804 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:20.114+0000 7f11053e2640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11001113e0 con 0x7f1100101d00 2024-06-18T02:30:23.804 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:20.114+0000 7f10feffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1100101d00 0x7f1100110ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:30:23.804 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:20.114+0000 7f10feffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1100101d00 0x7f1100110ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34914/0 (socket says 172.21.15.1:34914) 2024-06-18T02:30:23.804 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:20.114+0000 7f10feffd640 1 -- 172.21.15.1:0/1214209948 learned_addr learned my addr 172.21.15.1:0/1214209948 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:30:23.804 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:20.114+0000 7f10feffd640 1 -- 172.21.15.1:0/1214209948 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10f0009450 con 0x7f1100101d00 2024-06-18T02:30:23.804 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:20.115+0000 7f10feffd640 1 --2- 172.21.15.1:0/1214209948 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1100101d00 0x7f1100110ea0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f10f00317b0 tx=0x7f10f0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:30:23.804 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:20.115+0000 7f10e77fe640 1 -- 172.21.15.1:0/1214209948 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f10f0031e90 con 0x7f1100101d00 2024-06-18T02:30:23.804 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:20.115+0000 7f11053e2640 1 -- 172.21.15.1:0/1214209948 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11001115e0 con 0x7f1100101d00 2024-06-18T02:30:23.804 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:20.115+0000 7f10e77fe640 1 -- 172.21.15.1:0/1214209948 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f10f003daf0 con 0x7f1100101d00 2024-06-18T02:30:23.804 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:20.115+0000 7f10e77fe640 1 -- 172.21.15.1:0/1214209948 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f10f003baa0 con 0x7f1100101d00 2024-06-18T02:30:23.805 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:20.115+0000 7f11053e2640 1 -- 172.21.15.1:0/1214209948 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1100111fe0 con 0x7f1100101d00 2024-06-18T02:30:23.805 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:20.116+0000 7f10e77fe640 1 -- 172.21.15.1:0/1214209948 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 4) v1 ==== 41935+0+0 (secure 0 0 0) 0x7f10f0039070 con 0x7f1100101d00 2024-06-18T02:30:23.805 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:20.116+0000 7f10e77fe640 1 --2- 172.21.15.1:0/1214209948 >> [v2:172.21.15.1:6800/752464819,v1:172.21.15.1:6801/752464819] conn(0x7f10e80347c0 0x7f10e8036c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:23.805 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:20.116+0000 7f10e77fe640 1 -- 172.21.15.1:0/1214209948 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f10f0067600 con 0x7f1100101d00 2024-06-18T02:30:23.805 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:20.116+0000 7f10fe7fc640 1 --2- 172.21.15.1:0/1214209948 >> [v2:172.21.15.1:6800/752464819,v1:172.21.15.1:6801/752464819] conn(0x7f10e80347c0 0x7f10e8036c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:30:23.805 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:20.116+0000 7f11053e2640 1 -- 172.21.15.1:0/1214209948 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f110005ead0 con 0x7f1100101d00 2024-06-18T02:30:23.805 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:20.116+0000 7f10fe7fc640 1 --2- 172.21.15.1:0/1214209948 >> [v2:172.21.15.1:6800/752464819,v1:172.21.15.1:6801/752464819] conn(0x7f10e80347c0 0x7f10e8036c70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f10f4006fd0 tx=0x7f10f4006db0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:30:23.805 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:20.121+0000 7f10e77fe640 1 -- 172.21.15.1:0/1214209948 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f10f0041020 con 0x7f1100101d00 2024-06-18T02:30:23.805 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:20.295+0000 7f11053e2640 1 -- 172.21.15.1:0/1214209948 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f1100063ec0 con 0x7f1100101d00 2024-06-18T02:30:23.805 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:20.295+0000 7f10e77fe640 1 -- 172.21.15.1:0/1214209948 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7f10f0037070 con 0x7f1100101d00 2024-06-18T02:30:23.805 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:20.297+0000 7f11053e2640 1 -- 172.21.15.1:0/1214209948 >> [v2:172.21.15.1:6800/752464819,v1:172.21.15.1:6801/752464819] conn(0x7f10e80347c0 msgr2=0x7f10e8036c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:30:23.805 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:20.297+0000 7f11053e2640 1 --2- 172.21.15.1:0/1214209948 >> [v2:172.21.15.1:6800/752464819,v1:172.21.15.1:6801/752464819] conn(0x7f10e80347c0 0x7f10e8036c70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f10f4006fd0 tx=0x7f10f4006db0 comp rx=0 tx=0).stop 2024-06-18T02:30:23.805 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:20.297+0000 7f11053e2640 1 -- 172.21.15.1:0/1214209948 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1100101d00 msgr2=0x7f1100110ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:30:23.806 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:20.297+0000 7f11053e2640 1 --2- 172.21.15.1:0/1214209948 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1100101d00 0x7f1100110ea0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f10f00317b0 tx=0x7f10f0008040 comp rx=0 tx=0).stop 2024-06-18T02:30:23.806 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:20.297+0000 7f11053e2640 1 -- 172.21.15.1:0/1214209948 shutdown_connections 2024-06-18T02:30:23.806 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:20.297+0000 7f11053e2640 1 --2- 172.21.15.1:0/1214209948 >> [v2:172.21.15.1:6800/752464819,v1:172.21.15.1:6801/752464819] conn(0x7f10e80347c0 0x7f10e8036c70 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:23.806 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:20.297+0000 7f11053e2640 1 --2- 172.21.15.1:0/1214209948 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1100101d00 0x7f1100110ea0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:23.806 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:20.297+0000 7f11053e2640 1 -- 172.21.15.1:0/1214209948 >> 172.21.15.1:0/1214209948 conn(0x7f11000fd500 msgr2=0x7f11000fe200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:30:23.806 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:20.297+0000 7f11053e2640 1 -- 172.21.15.1:0/1214209948 shutdown_connections 2024-06-18T02:30:23.806 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:20.297+0000 7f11053e2640 1 -- 172.21.15.1:0/1214209948 wait complete. 2024-06-18T02:30:23.806 INFO:teuthology.orchestra.run.smithi001.stdout:Enabling cephadm module... 2024-06-18T02:30:24.787 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:24 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/3107266376' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-06-18T02:30:25.096 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:24.231+0000 7fe56635c640 1 Processor -- start 2024-06-18T02:30:25.096 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:24.231+0000 7fe56635c640 1 -- start start 2024-06-18T02:30:25.097 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:24.231+0000 7fe56635c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe560101c80 0x7fe560102090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:25.097 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:24.231+0000 7fe56635c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5601025d0 con 0x7fe560101c80 2024-06-18T02:30:25.097 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:24.232+0000 7fe55ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe560101c80 0x7fe560102090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:30:25.097 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:24.232+0000 7fe55ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe560101c80 0x7fe560102090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43342/0 (socket says 172.21.15.1:43342) 2024-06-18T02:30:25.097 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:24.232+0000 7fe55ffff640 1 -- 172.21.15.1:0/2499016384 learned_addr learned my addr 172.21.15.1:0/2499016384 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:30:25.097 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:24.232+0000 7fe55ffff640 1 -- 172.21.15.1:0/2499016384 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe560102710 con 0x7fe560101c80 2024-06-18T02:30:25.097 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:24.233+0000 7fe55ffff640 1 --2- 172.21.15.1:0/2499016384 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe560101c80 0x7fe560102090 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe550009840 tx=0x7fe5500091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a71702f31352632 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:30:25.097 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:24.233+0000 7fe55effd640 1 -- 172.21.15.1:0/2499016384 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe550032440 con 0x7fe560101c80 2024-06-18T02:30:25.097 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:24.233+0000 7fe55effd640 1 -- 172.21.15.1:0/2499016384 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fe550032a30 con 0x7fe560101c80 2024-06-18T02:30:25.097 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:24.233+0000 7fe55effd640 1 -- 172.21.15.1:0/2499016384 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe550031ad0 con 0x7fe560101c80 2024-06-18T02:30:25.097 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:24.234+0000 7fe56635c640 1 -- 172.21.15.1:0/2499016384 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe560101c80 msgr2=0x7fe560102090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:30:25.097 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:24.234+0000 7fe56635c640 1 --2- 172.21.15.1:0/2499016384 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe560101c80 0x7fe560102090 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe550009840 tx=0x7fe5500091d0 comp rx=0 tx=0).stop 2024-06-18T02:30:25.097 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:24.234+0000 7fe56635c640 1 -- 172.21.15.1:0/2499016384 shutdown_connections 2024-06-18T02:30:25.098 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:24.234+0000 7fe56635c640 1 --2- 172.21.15.1:0/2499016384 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe560101c80 0x7fe560102090 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:25.098 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:24.234+0000 7fe56635c640 1 -- 172.21.15.1:0/2499016384 >> 172.21.15.1:0/2499016384 conn(0x7fe5600fd440 msgr2=0x7fe5600ff870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:30:25.098 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:24.234+0000 7fe56635c640 1 -- 172.21.15.1:0/2499016384 shutdown_connections 2024-06-18T02:30:25.098 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:24.234+0000 7fe56635c640 1 -- 172.21.15.1:0/2499016384 wait complete. 2024-06-18T02:30:25.098 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:24.234+0000 7fe56635c640 1 Processor -- start 2024-06-18T02:30:25.098 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:24.234+0000 7fe56635c640 1 -- start start 2024-06-18T02:30:25.098 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:24.235+0000 7fe56635c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe560101c80 0x7fe560110aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:25.098 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:24.235+0000 7fe56635c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe560110fe0 con 0x7fe560101c80 2024-06-18T02:30:25.098 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:24.235+0000 7fe55ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe560101c80 0x7fe560110aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:30:25.098 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:24.235+0000 7fe55ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe560101c80 0x7fe560110aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43352/0 (socket says 172.21.15.1:43352) 2024-06-18T02:30:25.098 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:24.235+0000 7fe55ffff640 1 -- 172.21.15.1:0/3107266376 learned_addr learned my addr 172.21.15.1:0/3107266376 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:30:25.098 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:24.235+0000 7fe55ffff640 1 -- 172.21.15.1:0/3107266376 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe550009450 con 0x7fe560101c80 2024-06-18T02:30:25.098 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:24.235+0000 7fe55ffff640 1 --2- 172.21.15.1:0/3107266376 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe560101c80 0x7fe560110aa0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe550009420 tx=0x7fe550008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:30:25.099 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:24.235+0000 7fe55cff9640 1 -- 172.21.15.1:0/3107266376 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe55003d5b0 con 0x7fe560101c80 2024-06-18T02:30:25.099 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:24.235+0000 7fe55cff9640 1 -- 172.21.15.1:0/3107266376 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fe55003dba0 con 0x7fe560101c80 2024-06-18T02:30:25.099 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:24.235+0000 7fe55cff9640 1 -- 172.21.15.1:0/3107266376 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe55003bb40 con 0x7fe560101c80 2024-06-18T02:30:25.099 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:24.235+0000 7fe56635c640 1 -- 172.21.15.1:0/3107266376 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5601111e0 con 0x7fe560101c80 2024-06-18T02:30:25.099 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:24.236+0000 7fe56635c640 1 -- 172.21.15.1:0/3107266376 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe560111650 con 0x7fe560101c80 2024-06-18T02:30:25.099 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:24.236+0000 7fe55cff9640 1 -- 172.21.15.1:0/3107266376 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 4) v1 ==== 41935+0+0 (secure 0 0 0) 0x7fe550039070 con 0x7fe560101c80 2024-06-18T02:30:25.099 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:24.237+0000 7fe55cff9640 1 --2- 172.21.15.1:0/3107266376 >> [v2:172.21.15.1:6800/752464819,v1:172.21.15.1:6801/752464819] conn(0x7fe5480347c0 0x7fe548036c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:25.099 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:24.237+0000 7fe55cff9640 1 -- 172.21.15.1:0/3107266376 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fe5500673c0 con 0x7fe560101c80 2024-06-18T02:30:25.099 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:24.237+0000 7fe55f7fe640 1 --2- 172.21.15.1:0/3107266376 >> [v2:172.21.15.1:6800/752464819,v1:172.21.15.1:6801/752464819] conn(0x7fe5480347c0 0x7fe548036c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:30:25.099 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:24.237+0000 7fe56635c640 1 -- 172.21.15.1:0/3107266376 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe56005ead0 con 0x7fe560101c80 2024-06-18T02:30:25.099 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:24.237+0000 7fe55f7fe640 1 --2- 172.21.15.1:0/3107266376 >> [v2:172.21.15.1:6800/752464819,v1:172.21.15.1:6801/752464819] conn(0x7fe5480347c0 0x7fe548036c70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe554006fd0 tx=0x7fe554006db0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:30:25.099 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:24.242+0000 7fe55cff9640 1 -- 172.21.15.1:0/3107266376 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fe550037020 con 0x7fe560101c80 2024-06-18T02:30:25.099 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:24.461+0000 7fe56635c640 1 -- 172.21.15.1:0/3107266376 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7fe56004b480 con 0x7fe560101c80 2024-06-18T02:30:25.100 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:24.514+0000 7fe55cff9640 1 -- 172.21.15.1:0/3107266376 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 5) v1 ==== 41946+0+0 (secure 0 0 0) 0x7fe550066220 con 0x7fe560101c80 2024-06-18T02:30:25.100 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:24.515+0000 7fe55cff9640 1 -- 172.21.15.1:0/3107266376 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7fe5500457c0 con 0x7fe560101c80 2024-06-18T02:30:25.100 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:24.518+0000 7fe56635c640 1 -- 172.21.15.1:0/3107266376 >> [v2:172.21.15.1:6800/752464819,v1:172.21.15.1:6801/752464819] conn(0x7fe5480347c0 msgr2=0x7fe548036c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:30:25.100 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:24.518+0000 7fe56635c640 1 --2- 172.21.15.1:0/3107266376 >> [v2:172.21.15.1:6800/752464819,v1:172.21.15.1:6801/752464819] conn(0x7fe5480347c0 0x7fe548036c70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe554006fd0 tx=0x7fe554006db0 comp rx=0 tx=0).stop 2024-06-18T02:30:25.100 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:24.518+0000 7fe56635c640 1 -- 172.21.15.1:0/3107266376 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe560101c80 msgr2=0x7fe560110aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:30:25.100 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:24.518+0000 7fe56635c640 1 --2- 172.21.15.1:0/3107266376 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe560101c80 0x7fe560110aa0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe550009420 tx=0x7fe550008040 comp rx=0 tx=0).stop 2024-06-18T02:30:25.100 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:24.518+0000 7fe56635c640 1 -- 172.21.15.1:0/3107266376 shutdown_connections 2024-06-18T02:30:25.100 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:24.518+0000 7fe56635c640 1 --2- 172.21.15.1:0/3107266376 >> [v2:172.21.15.1:6800/752464819,v1:172.21.15.1:6801/752464819] conn(0x7fe5480347c0 0x7fe548036c70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:25.100 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:24.518+0000 7fe56635c640 1 --2- 172.21.15.1:0/3107266376 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe560101c80 0x7fe560110aa0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:25.100 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:24.518+0000 7fe56635c640 1 -- 172.21.15.1:0/3107266376 >> 172.21.15.1:0/3107266376 conn(0x7fe5600fd440 msgr2=0x7fe5600fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:30:25.100 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:24.519+0000 7fe56635c640 1 -- 172.21.15.1:0/3107266376 shutdown_connections 2024-06-18T02:30:25.100 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:24.519+0000 7fe56635c640 1 -- 172.21.15.1:0/3107266376 wait complete. 2024-06-18T02:30:25.627 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:25 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/3107266376' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-06-18T02:30:25.627 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:25 smithi001 ceph-mon[29099]: mgrmap e5: smithi001.teoawz(active, since 14s) 2024-06-18T02:30:26.410 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout { 2024-06-18T02:30:26.410 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-06-18T02:30:26.410 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "available": true, 2024-06-18T02:30:26.410 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "active_name": "smithi001.teoawz", 2024-06-18T02:30:26.410 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-06-18T02:30:26.410 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout } 2024-06-18T02:30:26.411 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:25.537+0000 7f3596b39640 1 Processor -- start 2024-06-18T02:30:26.411 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:25.537+0000 7f3596b39640 1 -- start start 2024-06-18T02:30:26.411 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:25.537+0000 7f3596b39640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3590100100 0x7f3590100510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:26.411 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:25.537+0000 7f3596b39640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3590100a50 con 0x7f3590100100 2024-06-18T02:30:26.411 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:25.537+0000 7f35948d5640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3590100100 0x7f3590100510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:30:26.411 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:25.538+0000 7f35948d5640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3590100100 0x7f3590100510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43382/0 (socket says 172.21.15.1:43382) 2024-06-18T02:30:26.411 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:25.538+0000 7f35948d5640 1 -- 172.21.15.1:0/4112164164 learned_addr learned my addr 172.21.15.1:0/4112164164 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:30:26.411 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:25.538+0000 7f35948d5640 1 -- 172.21.15.1:0/4112164164 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3590100b90 con 0x7f3590100100 2024-06-18T02:30:26.411 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:25.538+0000 7f35948d5640 1 --2- 172.21.15.1:0/4112164164 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3590100100 0x7f3590100510 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3584009840 tx=0x7f35840091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b9d4d1ff9f0093a server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:30:26.411 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:25.539+0000 7f358f7fe640 1 -- 172.21.15.1:0/4112164164 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3584032440 con 0x7f3590100100 2024-06-18T02:30:26.411 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:25.539+0000 7f358f7fe640 1 -- 172.21.15.1:0/4112164164 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f3584032a30 con 0x7f3590100100 2024-06-18T02:30:26.411 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:25.539+0000 7f358f7fe640 1 -- 172.21.15.1:0/4112164164 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3584031ad0 con 0x7f3590100100 2024-06-18T02:30:26.411 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:25.539+0000 7f3596b39640 1 -- 172.21.15.1:0/4112164164 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3590100100 msgr2=0x7f3590100510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:30:26.412 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:25.539+0000 7f3596b39640 1 --2- 172.21.15.1:0/4112164164 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3590100100 0x7f3590100510 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3584009840 tx=0x7f35840091d0 comp rx=0 tx=0).stop 2024-06-18T02:30:26.412 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:25.539+0000 7f3596b39640 1 -- 172.21.15.1:0/4112164164 shutdown_connections 2024-06-18T02:30:26.412 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:25.539+0000 7f3596b39640 1 --2- 172.21.15.1:0/4112164164 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3590100100 0x7f3590100510 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:26.412 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:25.539+0000 7f3596b39640 1 -- 172.21.15.1:0/4112164164 >> 172.21.15.1:0/4112164164 conn(0x7f35900fbba0 msgr2=0x7f35900fdfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:30:26.412 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:25.540+0000 7f3596b39640 1 -- 172.21.15.1:0/4112164164 shutdown_connections 2024-06-18T02:30:26.412 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:25.540+0000 7f3596b39640 1 -- 172.21.15.1:0/4112164164 wait complete. 2024-06-18T02:30:26.412 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:25.540+0000 7f3596b39640 1 Processor -- start 2024-06-18T02:30:26.412 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:25.540+0000 7f3596b39640 1 -- start start 2024-06-18T02:30:26.412 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:25.540+0000 7f3596b39640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3590100100 0x7f3590110ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:26.412 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:25.540+0000 7f3596b39640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3590110ff0 con 0x7f3590100100 2024-06-18T02:30:26.412 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:25.541+0000 7f35948d5640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3590100100 0x7f3590110ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:30:26.412 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:25.541+0000 7f35948d5640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3590100100 0x7f3590110ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43394/0 (socket says 172.21.15.1:43394) 2024-06-18T02:30:26.412 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:25.541+0000 7f35948d5640 1 -- 172.21.15.1:0/381102043 learned_addr learned my addr 172.21.15.1:0/381102043 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:30:26.413 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:25.541+0000 7f35948d5640 1 -- 172.21.15.1:0/381102043 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3584009450 con 0x7f3590100100 2024-06-18T02:30:26.413 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:25.541+0000 7f35948d5640 1 --2- 172.21.15.1:0/381102043 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3590100100 0x7f3590110ab0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3584009420 tx=0x7f3584008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:30:26.413 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:25.541+0000 7f358d7fa640 1 -- 172.21.15.1:0/381102043 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f358403d5b0 con 0x7f3590100100 2024-06-18T02:30:26.413 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:25.541+0000 7f358d7fa640 1 -- 172.21.15.1:0/381102043 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f358403dba0 con 0x7f3590100100 2024-06-18T02:30:26.413 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:25.541+0000 7f3596b39640 1 -- 172.21.15.1:0/381102043 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35901111f0 con 0x7f3590100100 2024-06-18T02:30:26.413 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:25.541+0000 7f358d7fa640 1 -- 172.21.15.1:0/381102043 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f358403bb40 con 0x7f3590100100 2024-06-18T02:30:26.413 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:25.541+0000 7f3596b39640 1 -- 172.21.15.1:0/381102043 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3590111660 con 0x7f3590100100 2024-06-18T02:30:26.413 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:25.542+0000 7f3596b39640 1 -- 172.21.15.1:0/381102043 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f359004b480 con 0x7f3590100100 2024-06-18T02:30:26.413 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:25.543+0000 7f358d7fa640 1 -- 172.21.15.1:0/381102043 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f3584039070 con 0x7f3590100100 2024-06-18T02:30:26.413 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:25.543+0000 7f358d7fa640 1 --2- 172.21.15.1:0/381102043 >> [v2:172.21.15.1:6800/752464819,v1:172.21.15.1:6801/752464819] conn(0x7f3580034860 0x7f3580036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:26.413 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:25.543+0000 7f358d7fa640 1 -- 172.21.15.1:0/381102043 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f35840672f0 con 0x7f3590100100 2024-06-18T02:30:26.413 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:25.543+0000 7f358ffff640 1 -- 172.21.15.1:0/381102043 >> [v2:172.21.15.1:6800/752464819,v1:172.21.15.1:6801/752464819] conn(0x7f3580034860 msgr2=0x7f3580036d10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/752464819 2024-06-18T02:30:26.414 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:25.543+0000 7f358ffff640 1 --2- 172.21.15.1:0/381102043 >> [v2:172.21.15.1:6800/752464819,v1:172.21.15.1:6801/752464819] conn(0x7f3580034860 0x7f3580036d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-18T02:30:26.414 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:25.547+0000 7f358d7fa640 1 -- 172.21.15.1:0/381102043 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f358403ca00 con 0x7f3590100100 2024-06-18T02:30:26.414 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:25.744+0000 7f358ffff640 1 -- 172.21.15.1:0/381102043 >> [v2:172.21.15.1:6800/752464819,v1:172.21.15.1:6801/752464819] conn(0x7f3580034860 msgr2=0x7f3580036d10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/752464819 2024-06-18T02:30:26.414 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:25.744+0000 7f358ffff640 1 --2- 172.21.15.1:0/381102043 >> [v2:172.21.15.1:6800/752464819,v1:172.21.15.1:6801/752464819] conn(0x7f3580034860 0x7f3580036d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-18T02:30:26.414 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:25.767+0000 7f3596b39640 1 -- 172.21.15.1:0/381102043 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f3590105cd0 con 0x7f3590100100 2024-06-18T02:30:26.414 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:25.768+0000 7f358d7fa640 1 -- 172.21.15.1:0/381102043 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7f3584041020 con 0x7f3590100100 2024-06-18T02:30:26.414 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:25.770+0000 7f3596b39640 1 -- 172.21.15.1:0/381102043 >> [v2:172.21.15.1:6800/752464819,v1:172.21.15.1:6801/752464819] conn(0x7f3580034860 msgr2=0x7f3580036d10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-18T02:30:26.414 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:25.770+0000 7f3596b39640 1 --2- 172.21.15.1:0/381102043 >> [v2:172.21.15.1:6800/752464819,v1:172.21.15.1:6801/752464819] conn(0x7f3580034860 0x7f3580036d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:26.414 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:25.770+0000 7f3596b39640 1 -- 172.21.15.1:0/381102043 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3590100100 msgr2=0x7f3590110ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:30:26.414 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:25.770+0000 7f3596b39640 1 --2- 172.21.15.1:0/381102043 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3590100100 0x7f3590110ab0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3584009420 tx=0x7f3584008040 comp rx=0 tx=0).stop 2024-06-18T02:30:26.414 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:25.770+0000 7f3596b39640 1 -- 172.21.15.1:0/381102043 shutdown_connections 2024-06-18T02:30:26.414 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:25.770+0000 7f3596b39640 1 --2- 172.21.15.1:0/381102043 >> [v2:172.21.15.1:6800/752464819,v1:172.21.15.1:6801/752464819] conn(0x7f3580034860 0x7f3580036d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:26.414 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:25.770+0000 7f3596b39640 1 --2- 172.21.15.1:0/381102043 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3590100100 0x7f3590110ab0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:26.415 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:25.770+0000 7f3596b39640 1 -- 172.21.15.1:0/381102043 >> 172.21.15.1:0/381102043 conn(0x7f35900fbba0 msgr2=0x7f35900fc6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:30:26.415 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:25.770+0000 7f3596b39640 1 -- 172.21.15.1:0/381102043 shutdown_connections 2024-06-18T02:30:26.415 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:25.770+0000 7f3596b39640 1 -- 172.21.15.1:0/381102043 wait complete. 2024-06-18T02:30:26.415 INFO:teuthology.orchestra.run.smithi001.stdout:Waiting for the mgr to restart... 2024-06-18T02:30:26.415 INFO:teuthology.orchestra.run.smithi001.stdout:Waiting for mgr epoch 5... 2024-06-18T02:30:26.622 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:26 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/381102043' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-06-18T02:30:29.377 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:29 smithi001 ceph-mon[29099]: Active manager daemon smithi001.teoawz restarted 2024-06-18T02:30:29.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:29 smithi001 ceph-mon[29099]: Activating manager daemon smithi001.teoawz 2024-06-18T02:30:29.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:29 smithi001 ceph-mon[29099]: osdmap e2: 0 total, 0 up, 0 in 2024-06-18T02:30:29.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:29 smithi001 ceph-mon[29099]: mgrmap e6: smithi001.teoawz(active, starting, since 0.00326944s) 2024-06-18T02:30:29.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:29 smithi001 ceph-mon[29099]: from='mgr.14118 172.21.15.1:0/2084661558' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "mon metadata", "id": "smithi001"}]: dispatch 2024-06-18T02:30:29.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:29 smithi001 ceph-mon[29099]: from='mgr.14118 172.21.15.1:0/2084661558' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "mgr metadata", "who": "smithi001.teoawz", "id": "smithi001.teoawz"}]: dispatch 2024-06-18T02:30:29.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:29 smithi001 ceph-mon[29099]: from='mgr.14118 172.21.15.1:0/2084661558' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-18T02:30:29.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:29 smithi001 ceph-mon[29099]: from='mgr.14118 172.21.15.1:0/2084661558' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-18T02:30:29.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:29 smithi001 ceph-mon[29099]: from='mgr.14118 172.21.15.1:0/2084661558' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-18T02:30:29.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:29 smithi001 ceph-mon[29099]: Manager daemon smithi001.teoawz is now available 2024-06-18T02:30:29.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:29 smithi001 ceph-mon[29099]: from='mgr.14118 172.21.15.1:0/2084661558' entity='mgr.smithi001.teoawz' 2024-06-18T02:30:29.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:29 smithi001 ceph-mon[29099]: from='mgr.14118 172.21.15.1:0/2084661558' entity='mgr.smithi001.teoawz' 2024-06-18T02:30:29.379 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:29 smithi001 ceph-mon[29099]: from='mgr.14118 172.21.15.1:0/2084661558' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:30:29.379 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:29 smithi001 ceph-mon[29099]: from='mgr.14118 172.21.15.1:0/2084661558' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:30:29.379 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:29 smithi001 ceph-mon[29099]: from='mgr.14118 172.21.15.1:0/2084661558' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:30:29.379 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:29 smithi001 ceph-mon[29099]: from='mgr.14118 172.21.15.1:0/2084661558' entity='mgr.smithi001.teoawz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi001.teoawz/mirror_snapshot_schedule"}]: dispatch 2024-06-18T02:30:29.379 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:29 smithi001 ceph-mon[29099]: from='mgr.14118 172.21.15.1:0/2084661558' entity='mgr.smithi001.teoawz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi001.teoawz/trash_purge_schedule"}]: dispatch 2024-06-18T02:30:30.525 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:30 smithi001 ceph-mon[29099]: Found migration_current of "None". Setting to last migration. 2024-06-18T02:30:30.526 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:30 smithi001 ceph-mon[29099]: mgrmap e7: smithi001.teoawz(active, since 1.00535s) 2024-06-18T02:30:30.795 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout { 2024-06-18T02:30:30.802 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-06-18T02:30:30.802 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "initialized": true 2024-06-18T02:30:30.802 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout } 2024-06-18T02:30:30.802 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:26.818+0000 7fcff6089640 1 Processor -- start 2024-06-18T02:30:30.802 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:26.818+0000 7fcff6089640 1 -- start start 2024-06-18T02:30:30.802 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:26.818+0000 7fcff6089640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcff0075bd0 0x7fcff0104990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:30.802 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:26.818+0000 7fcff6089640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcff0104ed0 con 0x7fcff0075bd0 2024-06-18T02:30:30.802 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:26.818+0000 7fcfef7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcff0075bd0 0x7fcff0104990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:30:30.803 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:26.818+0000 7fcfef7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcff0075bd0 0x7fcff0104990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43410/0 (socket says 172.21.15.1:43410) 2024-06-18T02:30:30.803 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:26.818+0000 7fcfef7fe640 1 -- 172.21.15.1:0/3222881645 learned_addr learned my addr 172.21.15.1:0/3222881645 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:30:30.803 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:26.819+0000 7fcfef7fe640 1 -- 172.21.15.1:0/3222881645 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcff0105010 con 0x7fcff0075bd0 2024-06-18T02:30:30.803 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:26.819+0000 7fcfef7fe640 1 --2- 172.21.15.1:0/3222881645 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcff0075bd0 0x7fcff0104990 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fcfe0009840 tx=0x7fcfe00091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d98d549c3529fa62 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:30:30.803 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:26.819+0000 7fcfee7fc640 1 -- 172.21.15.1:0/3222881645 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcfe0032440 con 0x7fcff0075bd0 2024-06-18T02:30:30.803 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:26.820+0000 7fcfee7fc640 1 -- 172.21.15.1:0/3222881645 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fcfe0032a30 con 0x7fcff0075bd0 2024-06-18T02:30:30.803 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:26.820+0000 7fcfee7fc640 1 -- 172.21.15.1:0/3222881645 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcfe0031ad0 con 0x7fcff0075bd0 2024-06-18T02:30:30.803 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:26.820+0000 7fcff6089640 1 -- 172.21.15.1:0/3222881645 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcff0075bd0 msgr2=0x7fcff0104990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:30:30.803 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:26.820+0000 7fcff6089640 1 --2- 172.21.15.1:0/3222881645 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcff0075bd0 0x7fcff0104990 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fcfe0009840 tx=0x7fcfe00091d0 comp rx=0 tx=0).stop 2024-06-18T02:30:30.803 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:26.820+0000 7fcff6089640 1 -- 172.21.15.1:0/3222881645 shutdown_connections 2024-06-18T02:30:30.803 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:26.820+0000 7fcff6089640 1 --2- 172.21.15.1:0/3222881645 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcff0075bd0 0x7fcff0104990 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:30.803 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:26.820+0000 7fcff6089640 1 -- 172.21.15.1:0/3222881645 >> 172.21.15.1:0/3222881645 conn(0x7fcff00fc4a0 msgr2=0x7fcff00fe8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:30:30.803 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:26.820+0000 7fcff6089640 1 -- 172.21.15.1:0/3222881645 shutdown_connections 2024-06-18T02:30:30.803 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:26.820+0000 7fcff6089640 1 -- 172.21.15.1:0/3222881645 wait complete. 2024-06-18T02:30:30.804 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:26.821+0000 7fcff6089640 1 Processor -- start 2024-06-18T02:30:30.804 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:26.821+0000 7fcff6089640 1 -- start start 2024-06-18T02:30:30.804 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:26.821+0000 7fcff6089640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcff0075bd0 0x7fcff0110b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:30.804 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:26.821+0000 7fcff6089640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcff0111080 con 0x7fcff0075bd0 2024-06-18T02:30:30.804 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:26.821+0000 7fcfef7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcff0075bd0 0x7fcff0110b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:30:30.804 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:26.821+0000 7fcfef7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcff0075bd0 0x7fcff0110b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43412/0 (socket says 172.21.15.1:43412) 2024-06-18T02:30:30.804 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:26.821+0000 7fcfef7fe640 1 -- 172.21.15.1:0/1964380106 learned_addr learned my addr 172.21.15.1:0/1964380106 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:30:30.804 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:26.822+0000 7fcfef7fe640 1 -- 172.21.15.1:0/1964380106 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfe0009450 con 0x7fcff0075bd0 2024-06-18T02:30:30.804 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:26.822+0000 7fcfef7fe640 1 --2- 172.21.15.1:0/1964380106 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcff0075bd0 0x7fcff0110b40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fcfe00317b0 tx=0x7fcfe0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:30:30.804 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:26.822+0000 7fcfd7fff640 1 -- 172.21.15.1:0/1964380106 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcfe0032890 con 0x7fcff0075bd0 2024-06-18T02:30:30.804 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:26.822+0000 7fcfd7fff640 1 -- 172.21.15.1:0/1964380106 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fcfe003d8e0 con 0x7fcff0075bd0 2024-06-18T02:30:30.804 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:26.822+0000 7fcff6089640 1 -- 172.21.15.1:0/1964380106 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcff0111280 con 0x7fcff0075bd0 2024-06-18T02:30:30.804 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:26.822+0000 7fcfd7fff640 1 -- 172.21.15.1:0/1964380106 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcfe003b9b0 con 0x7fcff0075bd0 2024-06-18T02:30:30.805 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:26.822+0000 7fcff6089640 1 -- 172.21.15.1:0/1964380106 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcff01116f0 con 0x7fcff0075bd0 2024-06-18T02:30:30.805 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:26.823+0000 7fcfd7fff640 1 -- 172.21.15.1:0/1964380106 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) v1 ==== 41946+0+0 (secure 0 0 0) 0x7fcfe0039070 con 0x7fcff0075bd0 2024-06-18T02:30:30.805 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:26.823+0000 7fcfd7fff640 1 --2- 172.21.15.1:0/1964380106 >> [v2:172.21.15.1:6800/752464819,v1:172.21.15.1:6801/752464819] conn(0x7fcfd8034860 0x7fcfd8036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:30.805 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:26.823+0000 7fcfd7fff640 1 -- 172.21.15.1:0/1964380106 --> [v2:172.21.15.1:6800/752464819,v1:172.21.15.1:6801/752464819] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcfd8037420 con 0x7fcfd8034860 2024-06-18T02:30:30.805 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:26.823+0000 7fcfd7fff640 1 -- 172.21.15.1:0/1964380106 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fcfe00673b0 con 0x7fcff0075bd0 2024-06-18T02:30:30.805 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:26.823+0000 7fcfeeffd640 1 -- 172.21.15.1:0/1964380106 >> [v2:172.21.15.1:6800/752464819,v1:172.21.15.1:6801/752464819] conn(0x7fcfd8034860 msgr2=0x7fcfd8036d10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/752464819 2024-06-18T02:30:30.805 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:26.823+0000 7fcfeeffd640 1 --2- 172.21.15.1:0/1964380106 >> [v2:172.21.15.1:6800/752464819,v1:172.21.15.1:6801/752464819] conn(0x7fcfd8034860 0x7fcfd8036d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-18T02:30:30.805 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:27.024+0000 7fcfeeffd640 1 -- 172.21.15.1:0/1964380106 >> [v2:172.21.15.1:6800/752464819,v1:172.21.15.1:6801/752464819] conn(0x7fcfd8034860 msgr2=0x7fcfd8036d10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/752464819 2024-06-18T02:30:30.805 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:27.024+0000 7fcfeeffd640 1 --2- 172.21.15.1:0/1964380106 >> [v2:172.21.15.1:6800/752464819,v1:172.21.15.1:6801/752464819] conn(0x7fcfd8034860 0x7fcfd8036d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-18T02:30:30.805 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:27.425+0000 7fcfeeffd640 1 -- 172.21.15.1:0/1964380106 >> [v2:172.21.15.1:6800/752464819,v1:172.21.15.1:6801/752464819] conn(0x7fcfd8034860 msgr2=0x7fcfd8036d10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/752464819 2024-06-18T02:30:30.805 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:27.425+0000 7fcfeeffd640 1 --2- 172.21.15.1:0/1964380106 >> [v2:172.21.15.1:6800/752464819,v1:172.21.15.1:6801/752464819] conn(0x7fcfd8034860 0x7fcfd8036d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-18T02:30:30.805 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:28.226+0000 7fcfeeffd640 1 -- 172.21.15.1:0/1964380106 >> [v2:172.21.15.1:6800/752464819,v1:172.21.15.1:6801/752464819] conn(0x7fcfd8034860 msgr2=0x7fcfd8036d10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/752464819 2024-06-18T02:30:30.805 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:28.226+0000 7fcfeeffd640 1 --2- 172.21.15.1:0/1964380106 >> [v2:172.21.15.1:6800/752464819,v1:172.21.15.1:6801/752464819] conn(0x7fcfd8034860 0x7fcfd8036d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-18T02:30:30.805 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:29.236+0000 7fcfd7fff640 1 -- 172.21.15.1:0/1964380106 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mgrmap(e 6) v1 ==== 41713+0+0 (secure 0 0 0) 0x7fcfe00668b0 con 0x7fcff0075bd0 2024-06-18T02:30:30.806 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:29.236+0000 7fcfd7fff640 1 -- 172.21.15.1:0/1964380106 >> [v2:172.21.15.1:6800/752464819,v1:172.21.15.1:6801/752464819] conn(0x7fcfd8034860 msgr2=0x7fcfd8036d10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-18T02:30:30.806 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:29.236+0000 7fcfd7fff640 1 --2- 172.21.15.1:0/1964380106 >> [v2:172.21.15.1:6800/752464819,v1:172.21.15.1:6801/752464819] conn(0x7fcfd8034860 0x7fcfd8036d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:30.806 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:30.237+0000 7fcfd7fff640 1 -- 172.21.15.1:0/1964380106 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 7) v1 ==== 41840+0+0 (secure 0 0 0) 0x7fcfe0066bb0 con 0x7fcff0075bd0 2024-06-18T02:30:30.806 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:30.238+0000 7fcfd7fff640 1 --2- 172.21.15.1:0/1964380106 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7fcfd8037fb0 0x7fcfd803a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:30.806 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:30.238+0000 7fcfd7fff640 1 -- 172.21.15.1:0/1964380106 --> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcfd8037420 con 0x7fcfd8037fb0 2024-06-18T02:30:30.806 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:30.238+0000 7fcfeeffd640 1 --2- 172.21.15.1:0/1964380106 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7fcfd8037fb0 0x7fcfd803a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:30:30.806 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:30.239+0000 7fcfeeffd640 1 --2- 172.21.15.1:0/1964380106 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7fcfd8037fb0 0x7fcfd803a390 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4003940 tx=0x7fcfe4009350 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:30:30.806 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:30.239+0000 7fcfd7fff640 1 -- 172.21.15.1:0/1964380106 <== mgr.14118 v2:172.21.15.1:6800/4268462134 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7fcfd8037420 con 0x7fcfd8037fb0 2024-06-18T02:30:30.806 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:30.245+0000 7fcff6089640 1 -- 172.21.15.1:0/1964380106 --> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fcff01116f0 con 0x7fcfd8037fb0 2024-06-18T02:30:30.806 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:30.245+0000 7fcfd7fff640 1 -- 172.21.15.1:0/1964380106 <== mgr.14118 v2:172.21.15.1:6800/4268462134 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fcff01116f0 con 0x7fcfd8037fb0 2024-06-18T02:30:30.806 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:30.245+0000 7fcff6089640 1 -- 172.21.15.1:0/1964380106 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7fcfd8037fb0 msgr2=0x7fcfd803a390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:30:30.806 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:30.245+0000 7fcff6089640 1 --2- 172.21.15.1:0/1964380106 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7fcfd8037fb0 0x7fcfd803a390 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4003940 tx=0x7fcfe4009350 comp rx=0 tx=0).stop 2024-06-18T02:30:30.806 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:30.245+0000 7fcff6089640 1 -- 172.21.15.1:0/1964380106 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcff0075bd0 msgr2=0x7fcff0110b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:30:30.807 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:30.245+0000 7fcff6089640 1 --2- 172.21.15.1:0/1964380106 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcff0075bd0 0x7fcff0110b40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fcfe00317b0 tx=0x7fcfe0008040 comp rx=0 tx=0).stop 2024-06-18T02:30:30.807 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:30.245+0000 7fcff6089640 1 -- 172.21.15.1:0/1964380106 shutdown_connections 2024-06-18T02:30:30.807 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:30.245+0000 7fcff6089640 1 --2- 172.21.15.1:0/1964380106 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7fcfd8037fb0 0x7fcfd803a390 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:30.807 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:30.245+0000 7fcff6089640 1 --2- 172.21.15.1:0/1964380106 >> [v2:172.21.15.1:6800/752464819,v1:172.21.15.1:6801/752464819] conn(0x7fcfd8034860 0x7fcfd8036d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:30.807 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:30.245+0000 7fcff6089640 1 --2- 172.21.15.1:0/1964380106 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcff0075bd0 0x7fcff0110b40 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:30.807 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:30.245+0000 7fcff6089640 1 -- 172.21.15.1:0/1964380106 >> 172.21.15.1:0/1964380106 conn(0x7fcff00fc4a0 msgr2=0x7fcff00fd040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:30:30.807 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:30.245+0000 7fcff6089640 1 -- 172.21.15.1:0/1964380106 shutdown_connections 2024-06-18T02:30:30.807 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:30.245+0000 7fcff6089640 1 -- 172.21.15.1:0/1964380106 wait complete. 2024-06-18T02:30:30.807 INFO:teuthology.orchestra.run.smithi001.stdout:mgr epoch 5 is available 2024-06-18T02:30:30.807 INFO:teuthology.orchestra.run.smithi001.stdout:Setting orchestrator backend to cephadm... 2024-06-18T02:30:31.377 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:31 smithi001 ceph-mon[29099]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-06-18T02:30:31.377 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:31 smithi001 ceph-mon[29099]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-06-18T02:30:31.981 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:31.192+0000 7f3a0c61d640 1 Processor -- start 2024-06-18T02:30:31.981 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:31.192+0000 7f3a0c61d640 1 -- start start 2024-06-18T02:30:31.981 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:31.192+0000 7f3a0c61d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a04075b80 0x7f3a04104980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:31.982 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:31.192+0000 7f3a0c61d640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a04075f90 con 0x7f3a04075b80 2024-06-18T02:30:31.982 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:31.192+0000 7f3a0a3b9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a04075b80 0x7f3a04104980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:30:31.982 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:31.192+0000 7f3a0a3b9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a04075b80 0x7f3a04104980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43494/0 (socket says 172.21.15.1:43494) 2024-06-18T02:30:31.982 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:31.192+0000 7f3a0a3b9640 1 -- 172.21.15.1:0/324515674 learned_addr learned my addr 172.21.15.1:0/324515674 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:30:31.982 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:31.192+0000 7f3a0a3b9640 1 -- 172.21.15.1:0/324515674 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a04104ec0 con 0x7f3a04075b80 2024-06-18T02:30:31.982 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:31.193+0000 7f3a0a3b9640 1 --2- 172.21.15.1:0/324515674 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a04075b80 0x7f3a04104980 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3a00009840 tx=0x7f3a000091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0d3e596abbfb749 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:30:31.982 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:31.193+0000 7f3a093b7640 1 -- 172.21.15.1:0/324515674 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a00032440 con 0x7f3a04075b80 2024-06-18T02:30:31.982 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:31.193+0000 7f3a093b7640 1 -- 172.21.15.1:0/324515674 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f3a00032a30 con 0x7f3a04075b80 2024-06-18T02:30:31.982 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:31.193+0000 7f3a093b7640 1 -- 172.21.15.1:0/324515674 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a00031ad0 con 0x7f3a04075b80 2024-06-18T02:30:31.982 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:31.194+0000 7f3a0c61d640 1 -- 172.21.15.1:0/324515674 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a04075b80 msgr2=0x7f3a04104980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:30:31.982 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:31.194+0000 7f3a0c61d640 1 --2- 172.21.15.1:0/324515674 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a04075b80 0x7f3a04104980 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3a00009840 tx=0x7f3a000091d0 comp rx=0 tx=0).stop 2024-06-18T02:30:31.983 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:31.194+0000 7f3a0c61d640 1 -- 172.21.15.1:0/324515674 shutdown_connections 2024-06-18T02:30:31.983 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:31.194+0000 7f3a0c61d640 1 --2- 172.21.15.1:0/324515674 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a04075b80 0x7f3a04104980 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:31.983 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:31.194+0000 7f3a0c61d640 1 -- 172.21.15.1:0/324515674 >> 172.21.15.1:0/324515674 conn(0x7f3a040fc450 msgr2=0x7f3a040fe880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:30:31.983 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:31.194+0000 7f3a0c61d640 1 -- 172.21.15.1:0/324515674 shutdown_connections 2024-06-18T02:30:31.983 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:31.194+0000 7f3a0c61d640 1 -- 172.21.15.1:0/324515674 wait complete. 2024-06-18T02:30:31.983 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:31.195+0000 7f3a0c61d640 1 Processor -- start 2024-06-18T02:30:31.983 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:31.195+0000 7f3a0c61d640 1 -- start start 2024-06-18T02:30:31.983 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:31.195+0000 7f3a0c61d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a04075b80 0x7f3a04110ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:31.983 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:31.195+0000 7f3a0c61d640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a04111020 con 0x7f3a04075b80 2024-06-18T02:30:31.983 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:31.195+0000 7f3a0a3b9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a04075b80 0x7f3a04110ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:30:31.983 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:31.195+0000 7f3a0a3b9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a04075b80 0x7f3a04110ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43504/0 (socket says 172.21.15.1:43504) 2024-06-18T02:30:31.983 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:31.195+0000 7f3a0a3b9640 1 -- 172.21.15.1:0/2358903035 learned_addr learned my addr 172.21.15.1:0/2358903035 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:30:31.983 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:31.195+0000 7f3a0a3b9640 1 -- 172.21.15.1:0/2358903035 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a00009450 con 0x7f3a04075b80 2024-06-18T02:30:31.983 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:31.195+0000 7f3a0a3b9640 1 --2- 172.21.15.1:0/2358903035 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a04075b80 0x7f3a04110ae0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f3a00006f00 tx=0x7f3a00008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:30:31.984 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:31.195+0000 7f39faffd640 1 -- 172.21.15.1:0/2358903035 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a0003d450 con 0x7f3a04075b80 2024-06-18T02:30:31.984 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:31.195+0000 7f39faffd640 1 -- 172.21.15.1:0/2358903035 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f3a0003da40 con 0x7f3a04075b80 2024-06-18T02:30:31.984 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:31.196+0000 7f3a0c61d640 1 -- 172.21.15.1:0/2358903035 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a04111220 con 0x7f3a04075b80 2024-06-18T02:30:31.984 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:31.196+0000 7f39faffd640 1 -- 172.21.15.1:0/2358903035 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a0003bb10 con 0x7f3a04075b80 2024-06-18T02:30:31.984 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:31.196+0000 7f3a0c61d640 1 -- 172.21.15.1:0/2358903035 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a04111690 con 0x7f3a04075b80 2024-06-18T02:30:31.984 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:31.196+0000 7f39faffd640 1 -- 172.21.15.1:0/2358903035 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 7) v1 ==== 41840+0+0 (secure 0 0 0) 0x7f3a00039070 con 0x7f3a04075b80 2024-06-18T02:30:31.984 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:31.197+0000 7f39faffd640 1 --2- 172.21.15.1:0/2358903035 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f39f00346e0 0x7f39f0036b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:31.984 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:31.197+0000 7f39faffd640 1 -- 172.21.15.1:0/2358903035 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f3a000673f0 con 0x7f3a04075b80 2024-06-18T02:30:31.984 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:31.197+0000 7f3a09bb8640 1 --2- 172.21.15.1:0/2358903035 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f39f00346e0 0x7f39f0036b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:30:31.984 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:31.197+0000 7f3a0c61d640 1 -- 172.21.15.1:0/2358903035 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a0405ead0 con 0x7f3a04075b80 2024-06-18T02:30:31.984 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:31.197+0000 7f3a09bb8640 1 --2- 172.21.15.1:0/2358903035 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f39f00346e0 0x7f39f0036b90 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f39f4006fd0 tx=0x7f39f4006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:30:31.984 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:31.202+0000 7f39faffd640 1 -- 172.21.15.1:0/2358903035 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f3a00041080 con 0x7f3a04075b80 2024-06-18T02:30:31.984 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:31.290+0000 7f39faffd640 1 -- 172.21.15.1:0/2358903035 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f3a00038280 con 0x7f3a04075b80 2024-06-18T02:30:31.984 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:31.411+0000 7f3a0c61d640 1 -- 172.21.15.1:0/2358903035 --> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f3a040fd7f0 con 0x7f39f00346e0 2024-06-18T02:30:31.985 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:31.419+0000 7f39faffd640 1 -- 172.21.15.1:0/2358903035 <== mgr.14118 v2:172.21.15.1:6800/4268462134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f3a040fd7f0 con 0x7f39f00346e0 2024-06-18T02:30:31.985 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:31.422+0000 7f3a0c61d640 1 -- 172.21.15.1:0/2358903035 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f39f00346e0 msgr2=0x7f39f0036b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:30:31.985 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:31.422+0000 7f3a0c61d640 1 --2- 172.21.15.1:0/2358903035 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f39f00346e0 0x7f39f0036b90 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f39f4006fd0 tx=0x7f39f4006db0 comp rx=0 tx=0).stop 2024-06-18T02:30:31.985 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:31.422+0000 7f3a0c61d640 1 -- 172.21.15.1:0/2358903035 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a04075b80 msgr2=0x7f3a04110ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:30:31.985 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:31.422+0000 7f3a0c61d640 1 --2- 172.21.15.1:0/2358903035 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a04075b80 0x7f3a04110ae0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f3a00006f00 tx=0x7f3a00008040 comp rx=0 tx=0).stop 2024-06-18T02:30:31.985 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:31.422+0000 7f3a0c61d640 1 -- 172.21.15.1:0/2358903035 shutdown_connections 2024-06-18T02:30:31.985 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:31.422+0000 7f3a0c61d640 1 --2- 172.21.15.1:0/2358903035 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f39f00346e0 0x7f39f0036b90 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:31.985 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:31.422+0000 7f3a0c61d640 1 --2- 172.21.15.1:0/2358903035 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a04075b80 0x7f3a04110ae0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:31.985 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:31.422+0000 7f3a0c61d640 1 -- 172.21.15.1:0/2358903035 >> 172.21.15.1:0/2358903035 conn(0x7f3a040fc450 msgr2=0x7f3a040fd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:30:31.985 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:31.422+0000 7f3a0c61d640 1 -- 172.21.15.1:0/2358903035 shutdown_connections 2024-06-18T02:30:31.985 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:31.422+0000 7f3a0c61d640 1 -- 172.21.15.1:0/2358903035 wait complete. 2024-06-18T02:30:32.553 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:32 smithi001 ceph-mon[29099]: mgrmap e8: smithi001.teoawz(active, since 2s) 2024-06-18T02:30:32.553 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:32 smithi001 ceph-mon[29099]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-06-18T02:30:32.554 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:32 smithi001 ceph-mon[29099]: from='mgr.14118 172.21.15.1:0/2084661558' entity='mgr.smithi001.teoawz' 2024-06-18T02:30:32.554 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:32 smithi001 ceph-mon[29099]: from='mgr.14118 172.21.15.1:0/2084661558' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:30:32.554 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:32 smithi001 ceph-mon[29099]: from='mgr.14118 172.21.15.1:0/2084661558' entity='mgr.smithi001.teoawz' 2024-06-18T02:30:32.554 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:32 smithi001 ceph-mon[29099]: from='mgr.14118 172.21.15.1:0/2084661558' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:30:33.122 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout value unchanged 2024-06-18T02:30:33.122 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:32.364+0000 7f01c6c7d640 1 Processor -- start 2024-06-18T02:30:33.122 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:32.364+0000 7f01c6c7d640 1 -- start start 2024-06-18T02:30:33.122 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:32.364+0000 7f01c6c7d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01c01031a0 0x7f01c01035b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:33.122 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:32.364+0000 7f01c6c7d640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01c0103b80 con 0x7f01c01031a0 2024-06-18T02:30:33.123 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:32.364+0000 7f01c4a19640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01c01031a0 0x7f01c01035b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:30:33.123 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:32.364+0000 7f01c4a19640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01c01031a0 0x7f01c01035b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43506/0 (socket says 172.21.15.1:43506) 2024-06-18T02:30:33.123 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:32.364+0000 7f01c4a19640 1 -- 172.21.15.1:0/4254092645 learned_addr learned my addr 172.21.15.1:0/4254092645 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:30:33.123 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:32.364+0000 7f01c4a19640 1 -- 172.21.15.1:0/4254092645 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01c0103cc0 con 0x7f01c01031a0 2024-06-18T02:30:33.123 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:32.365+0000 7f01c4a19640 1 --2- 172.21.15.1:0/4254092645 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01c01031a0 0x7f01c01035b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f01b4009840 tx=0x7f01b40091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb3870b88001cc10 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:30:33.123 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:32.365+0000 7f01bf7fe640 1 -- 172.21.15.1:0/4254092645 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f01b4032440 con 0x7f01c01031a0 2024-06-18T02:30:33.123 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:32.365+0000 7f01bf7fe640 1 -- 172.21.15.1:0/4254092645 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f01b4032a30 con 0x7f01c01031a0 2024-06-18T02:30:33.123 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:32.365+0000 7f01bf7fe640 1 -- 172.21.15.1:0/4254092645 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f01b4031ad0 con 0x7f01c01031a0 2024-06-18T02:30:33.123 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:32.366+0000 7f01c6c7d640 1 -- 172.21.15.1:0/4254092645 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01c01031a0 msgr2=0x7f01c01035b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:30:33.123 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:32.366+0000 7f01c6c7d640 1 --2- 172.21.15.1:0/4254092645 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01c01031a0 0x7f01c01035b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f01b4009840 tx=0x7f01b40091d0 comp rx=0 tx=0).stop 2024-06-18T02:30:33.123 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:32.366+0000 7f01c6c7d640 1 -- 172.21.15.1:0/4254092645 shutdown_connections 2024-06-18T02:30:33.123 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:32.366+0000 7f01c6c7d640 1 --2- 172.21.15.1:0/4254092645 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01c01031a0 0x7f01c01035b0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:33.123 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:32.366+0000 7f01c6c7d640 1 -- 172.21.15.1:0/4254092645 >> 172.21.15.1:0/4254092645 conn(0x7f01c00fbc10 msgr2=0x7f01c00fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:30:33.124 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:32.366+0000 7f01c6c7d640 1 -- 172.21.15.1:0/4254092645 shutdown_connections 2024-06-18T02:30:33.124 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:32.366+0000 7f01c6c7d640 1 -- 172.21.15.1:0/4254092645 wait complete. 2024-06-18T02:30:33.124 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:32.367+0000 7f01c6c7d640 1 Processor -- start 2024-06-18T02:30:33.124 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:32.367+0000 7f01c6c7d640 1 -- start start 2024-06-18T02:30:33.124 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:32.367+0000 7f01c6c7d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01c01031a0 0x7f01c0110b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:33.124 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:32.367+0000 7f01c6c7d640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01c0111070 con 0x7f01c01031a0 2024-06-18T02:30:33.124 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:32.367+0000 7f01c4a19640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01c01031a0 0x7f01c0110b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:30:33.124 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:32.367+0000 7f01c4a19640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01c01031a0 0x7f01c0110b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43512/0 (socket says 172.21.15.1:43512) 2024-06-18T02:30:33.124 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:32.367+0000 7f01c4a19640 1 -- 172.21.15.1:0/775429640 learned_addr learned my addr 172.21.15.1:0/775429640 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:30:33.124 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:32.367+0000 7f01c4a19640 1 -- 172.21.15.1:0/775429640 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01b4009450 con 0x7f01c01031a0 2024-06-18T02:30:33.124 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:32.368+0000 7f01c4a19640 1 --2- 172.21.15.1:0/775429640 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01c01031a0 0x7f01c0110b30 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f01b4031670 tx=0x7f01b4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:30:33.124 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:32.368+0000 7f01bd7fa640 1 -- 172.21.15.1:0/775429640 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f01b403d450 con 0x7f01c01031a0 2024-06-18T02:30:33.125 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:32.368+0000 7f01bd7fa640 1 -- 172.21.15.1:0/775429640 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f01b403da40 con 0x7f01c01031a0 2024-06-18T02:30:33.125 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:32.368+0000 7f01bd7fa640 1 -- 172.21.15.1:0/775429640 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f01b403ba00 con 0x7f01c01031a0 2024-06-18T02:30:33.125 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:32.368+0000 7f01c6c7d640 1 -- 172.21.15.1:0/775429640 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01c0111270 con 0x7f01c01031a0 2024-06-18T02:30:33.125 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:32.368+0000 7f01c6c7d640 1 -- 172.21.15.1:0/775429640 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01c01116e0 con 0x7f01c01031a0 2024-06-18T02:30:33.125 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:32.369+0000 7f01bd7fa640 1 -- 172.21.15.1:0/775429640 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f01b4039070 con 0x7f01c01031a0 2024-06-18T02:30:33.125 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:32.369+0000 7f01bd7fa640 1 --2- 172.21.15.1:0/775429640 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f01b0034860 0x7f01b0036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:33.125 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:32.369+0000 7f01c6c7d640 1 -- 172.21.15.1:0/775429640 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01c004b480 con 0x7f01c01031a0 2024-06-18T02:30:33.125 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:32.369+0000 7f01bd7fa640 1 -- 172.21.15.1:0/775429640 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f01b40675e0 con 0x7f01c01031a0 2024-06-18T02:30:33.125 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:32.369+0000 7f01bffff640 1 --2- 172.21.15.1:0/775429640 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f01b0034860 0x7f01b0036d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:30:33.125 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:32.369+0000 7f01bffff640 1 --2- 172.21.15.1:0/775429640 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f01b0034860 0x7f01b0036d10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f01ac006fd0 tx=0x7f01ac006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:30:33.125 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:32.373+0000 7f01bd7fa640 1 -- 172.21.15.1:0/775429640 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f01b4041080 con 0x7f01c01031a0 2024-06-18T02:30:33.125 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:32.553+0000 7f01c6c7d640 1 -- 172.21.15.1:0/775429640 --> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f01c00fcec0 con 0x7f01b0034860 2024-06-18T02:30:33.126 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:32.554+0000 7f01bd7fa640 1 -- 172.21.15.1:0/775429640 <== mgr.14118 v2:172.21.15.1:6800/4268462134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f01c00fcec0 con 0x7f01b0034860 2024-06-18T02:30:33.126 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:32.556+0000 7f01c6c7d640 1 -- 172.21.15.1:0/775429640 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f01b0034860 msgr2=0x7f01b0036d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:30:33.126 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:32.556+0000 7f01c6c7d640 1 --2- 172.21.15.1:0/775429640 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f01b0034860 0x7f01b0036d10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f01ac006fd0 tx=0x7f01ac006db0 comp rx=0 tx=0).stop 2024-06-18T02:30:33.126 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:32.556+0000 7f01c6c7d640 1 -- 172.21.15.1:0/775429640 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01c01031a0 msgr2=0x7f01c0110b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:30:33.126 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:32.556+0000 7f01c6c7d640 1 --2- 172.21.15.1:0/775429640 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01c01031a0 0x7f01c0110b30 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f01b4031670 tx=0x7f01b4008040 comp rx=0 tx=0).stop 2024-06-18T02:30:33.126 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:32.556+0000 7f01c6c7d640 1 -- 172.21.15.1:0/775429640 shutdown_connections 2024-06-18T02:30:33.126 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:32.556+0000 7f01c6c7d640 1 --2- 172.21.15.1:0/775429640 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f01b0034860 0x7f01b0036d10 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:33.126 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:32.556+0000 7f01c6c7d640 1 --2- 172.21.15.1:0/775429640 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f01c01031a0 0x7f01c0110b30 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:33.126 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:32.556+0000 7f01c6c7d640 1 -- 172.21.15.1:0/775429640 >> 172.21.15.1:0/775429640 conn(0x7f01c00fbc10 msgr2=0x7f01c00fc7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:30:33.126 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:32.556+0000 7f01c6c7d640 1 -- 172.21.15.1:0/775429640 shutdown_connections 2024-06-18T02:30:33.126 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:32.556+0000 7f01c6c7d640 1 -- 172.21.15.1:0/775429640 wait complete. 2024-06-18T02:30:33.127 INFO:teuthology.orchestra.run.smithi001.stdout:Generating ssh key... 2024-06-18T02:30:33.367 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:33 smithi001 ceph-mon[29099]: [18/Jun/2024:02:30:32] ENGINE Bus STARTING 2024-06-18T02:30:33.367 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:33 smithi001 ceph-mon[29099]: [18/Jun/2024:02:30:32] ENGINE Serving on https://172.21.15.1:7150 2024-06-18T02:30:33.367 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:33 smithi001 ceph-mon[29099]: [18/Jun/2024:02:30:32] ENGINE Bus STARTED 2024-06-18T02:30:33.368 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:33 smithi001 ceph-mon[29099]: [18/Jun/2024:02:30:32] ENGINE Error in HTTPServer.serve 2024-06-18T02:30:33.368 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Traceback (most recent call last): 2024-06-18T02:30:33.368 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-06-18T02:30:33.368 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: self._connections.run(self.expiration_interval) 2024-06-18T02:30:33.368 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-06-18T02:30:33.368 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: self._run(expiration_interval) 2024-06-18T02:30:33.368 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-06-18T02:30:33.368 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-06-18T02:30:33.368 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-06-18T02:30:33.368 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-06-18T02:30:33.368 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-06-18T02:30:33.368 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: s = self.context.wrap_socket( 2024-06-18T02:30:33.369 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-06-18T02:30:33.369 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: return self.sslsocket_class._create( 2024-06-18T02:30:33.369 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-06-18T02:30:33.369 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: self.do_handshake() 2024-06-18T02:30:33.369 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-06-18T02:30:33.369 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: self._sslobj.do_handshake() 2024-06-18T02:30:33.369 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1129) 2024-06-18T02:30:33.369 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:33 smithi001 ceph-mon[29099]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-06-18T02:30:34.429 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:33.538+0000 7f0cfaa33640 1 Processor -- start 2024-06-18T02:30:34.429 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:33.538+0000 7f0cfaa33640 1 -- start start 2024-06-18T02:30:34.429 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:33.538+0000 7f0cfaa33640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cf4075b20 0x7f0cf41049e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:34.429 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:33.539+0000 7f0cfaa33640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf4075f30 con 0x7f0cf4075b20 2024-06-18T02:30:34.429 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:33.539+0000 7f0cf3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cf4075b20 0x7f0cf41049e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:30:34.429 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:33.539+0000 7f0cf3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cf4075b20 0x7f0cf41049e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43528/0 (socket says 172.21.15.1:43528) 2024-06-18T02:30:34.429 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:33.539+0000 7f0cf3fff640 1 -- 172.21.15.1:0/2948226023 learned_addr learned my addr 172.21.15.1:0/2948226023 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:30:34.429 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:33.539+0000 7f0cf3fff640 1 -- 172.21.15.1:0/2948226023 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cf4104f20 con 0x7f0cf4075b20 2024-06-18T02:30:34.429 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:33.540+0000 7f0cf3fff640 1 --2- 172.21.15.1:0/2948226023 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cf4075b20 0x7f0cf41049e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0ce4009840 tx=0x7f0ce40091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=399857839ad29689 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:30:34.429 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:33.540+0000 7f0cf2ffd640 1 -- 172.21.15.1:0/2948226023 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ce4032440 con 0x7f0cf4075b20 2024-06-18T02:30:34.430 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:33.540+0000 7f0cf2ffd640 1 -- 172.21.15.1:0/2948226023 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f0ce4032a30 con 0x7f0cf4075b20 2024-06-18T02:30:34.430 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:33.540+0000 7f0cf2ffd640 1 -- 172.21.15.1:0/2948226023 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ce4031ad0 con 0x7f0cf4075b20 2024-06-18T02:30:34.430 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:33.541+0000 7f0cfaa33640 1 -- 172.21.15.1:0/2948226023 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cf4075b20 msgr2=0x7f0cf41049e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:30:34.430 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:33.541+0000 7f0cfaa33640 1 --2- 172.21.15.1:0/2948226023 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cf4075b20 0x7f0cf41049e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0ce4009840 tx=0x7f0ce40091d0 comp rx=0 tx=0).stop 2024-06-18T02:30:34.430 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:33.541+0000 7f0cfaa33640 1 -- 172.21.15.1:0/2948226023 shutdown_connections 2024-06-18T02:30:34.430 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:33.541+0000 7f0cfaa33640 1 --2- 172.21.15.1:0/2948226023 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cf4075b20 0x7f0cf41049e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:34.430 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:33.541+0000 7f0cfaa33640 1 -- 172.21.15.1:0/2948226023 >> 172.21.15.1:0/2948226023 conn(0x7f0cf40fc450 msgr2=0x7f0cf40fe880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:30:34.430 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:33.541+0000 7f0cfaa33640 1 -- 172.21.15.1:0/2948226023 shutdown_connections 2024-06-18T02:30:34.430 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:33.541+0000 7f0cfaa33640 1 -- 172.21.15.1:0/2948226023 wait complete. 2024-06-18T02:30:34.430 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:33.541+0000 7f0cfaa33640 1 Processor -- start 2024-06-18T02:30:34.430 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:33.541+0000 7f0cfaa33640 1 -- start start 2024-06-18T02:30:34.430 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:33.542+0000 7f0cfaa33640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cf4075b20 0x7f0cf4110ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:34.430 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:33.542+0000 7f0cfaa33640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf4111020 con 0x7f0cf4075b20 2024-06-18T02:30:34.431 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:33.542+0000 7f0cf3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cf4075b20 0x7f0cf4110ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:30:34.431 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:33.542+0000 7f0cf3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cf4075b20 0x7f0cf4110ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43544/0 (socket says 172.21.15.1:43544) 2024-06-18T02:30:34.431 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:33.542+0000 7f0cf3fff640 1 -- 172.21.15.1:0/4170573806 learned_addr learned my addr 172.21.15.1:0/4170573806 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:30:34.431 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:33.542+0000 7f0cf3fff640 1 -- 172.21.15.1:0/4170573806 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ce4009450 con 0x7f0cf4075b20 2024-06-18T02:30:34.431 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:33.542+0000 7f0cf3fff640 1 --2- 172.21.15.1:0/4170573806 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cf4075b20 0x7f0cf4110ae0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0ce4006f00 tx=0x7f0ce4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:30:34.431 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:33.542+0000 7f0cf0ff9640 1 -- 172.21.15.1:0/4170573806 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ce403d450 con 0x7f0cf4075b20 2024-06-18T02:30:34.431 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:33.543+0000 7f0cf0ff9640 1 -- 172.21.15.1:0/4170573806 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f0ce403da40 con 0x7f0cf4075b20 2024-06-18T02:30:34.431 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:33.543+0000 7f0cfaa33640 1 -- 172.21.15.1:0/4170573806 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cf4111220 con 0x7f0cf4075b20 2024-06-18T02:30:34.431 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:33.543+0000 7f0cf0ff9640 1 -- 172.21.15.1:0/4170573806 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ce403bb10 con 0x7f0cf4075b20 2024-06-18T02:30:34.431 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:33.543+0000 7f0cfaa33640 1 -- 172.21.15.1:0/4170573806 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cf4111690 con 0x7f0cf4075b20 2024-06-18T02:30:34.431 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:33.543+0000 7f0cf0ff9640 1 -- 172.21.15.1:0/4170573806 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f0ce4039070 con 0x7f0cf4075b20 2024-06-18T02:30:34.431 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:33.544+0000 7f0cf0ff9640 1 --2- 172.21.15.1:0/4170573806 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f0cdc034810 0x7f0cdc036cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:34.431 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:33.544+0000 7f0cf0ff9640 1 -- 172.21.15.1:0/4170573806 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f0ce4068610 con 0x7f0cf4075b20 2024-06-18T02:30:34.431 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:33.544+0000 7f0cf37fe640 1 --2- 172.21.15.1:0/4170573806 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f0cdc034810 0x7f0cdc036cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:30:34.432 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:33.544+0000 7f0cfaa33640 1 -- 172.21.15.1:0/4170573806 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cf405ead0 con 0x7f0cf4075b20 2024-06-18T02:30:34.432 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:33.544+0000 7f0cf37fe640 1 --2- 172.21.15.1:0/4170573806 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f0cdc034810 0x7f0cdc036cc0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0ce8006fd0 tx=0x7f0ce8006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:30:34.432 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:33.549+0000 7f0cf0ff9640 1 -- 172.21.15.1:0/4170573806 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f0ce4041020 con 0x7f0cf4075b20 2024-06-18T02:30:34.432 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:33.723+0000 7f0cfaa33640 1 -- 172.21.15.1:0/4170573806 --> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f0cf40fd7f0 con 0x7f0cdc034810 2024-06-18T02:30:34.432 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:33.849+0000 7f0cf0ff9640 1 -- 172.21.15.1:0/4170573806 <== mgr.14118 v2:172.21.15.1:6800/4268462134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f0cf40fd7f0 con 0x7f0cdc034810 2024-06-18T02:30:34.432 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:33.851+0000 7f0cfaa33640 1 -- 172.21.15.1:0/4170573806 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f0cdc034810 msgr2=0x7f0cdc036cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:30:34.432 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:33.851+0000 7f0cfaa33640 1 --2- 172.21.15.1:0/4170573806 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f0cdc034810 0x7f0cdc036cc0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0ce8006fd0 tx=0x7f0ce8006db0 comp rx=0 tx=0).stop 2024-06-18T02:30:34.432 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:33.851+0000 7f0cfaa33640 1 -- 172.21.15.1:0/4170573806 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cf4075b20 msgr2=0x7f0cf4110ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:30:34.432 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:33.851+0000 7f0cfaa33640 1 --2- 172.21.15.1:0/4170573806 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cf4075b20 0x7f0cf4110ae0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0ce4006f00 tx=0x7f0ce4008040 comp rx=0 tx=0).stop 2024-06-18T02:30:34.432 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:33.851+0000 7f0cfaa33640 1 -- 172.21.15.1:0/4170573806 shutdown_connections 2024-06-18T02:30:34.432 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:33.851+0000 7f0cfaa33640 1 --2- 172.21.15.1:0/4170573806 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f0cdc034810 0x7f0cdc036cc0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:34.432 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:33.851+0000 7f0cfaa33640 1 --2- 172.21.15.1:0/4170573806 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cf4075b20 0x7f0cf4110ae0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:34.432 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:33.851+0000 7f0cfaa33640 1 -- 172.21.15.1:0/4170573806 >> 172.21.15.1:0/4170573806 conn(0x7f0cf40fc450 msgr2=0x7f0cf40fd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:30:34.433 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:33.851+0000 7f0cfaa33640 1 -- 172.21.15.1:0/4170573806 shutdown_connections 2024-06-18T02:30:34.433 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:33.851+0000 7f0cfaa33640 1 -- 172.21.15.1:0/4170573806 wait complete. 2024-06-18T02:30:34.972 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:34 smithi001 ceph-mon[29099]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-06-18T02:30:34.972 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:34 smithi001 ceph-mon[29099]: Generating ssh key... 2024-06-18T02:30:34.972 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:34 smithi001 ceph-mon[29099]: from='mgr.14118 172.21.15.1:0/2084661558' entity='mgr.smithi001.teoawz' 2024-06-18T02:30:34.972 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:34 smithi001 ceph-mon[29099]: from='mgr.14118 172.21.15.1:0/2084661558' entity='mgr.smithi001.teoawz' 2024-06-18T02:30:35.548 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDVBZQGul2r4JowODwEUyOlWTzn0CqoGU0SOPOVPnFp/5+EFQlYXFGXpzHx6xz3BIkFoo/k86YCxPAT5+6kJwj98TvvCtchAgLk2IZ9oVdnXURVwDaL/4RVUyGle/8Z/mJ6uHvyO2eyK/jJPYjWhBPCUVx1ah0BoAi46VdBDLmnOSijEHLX3Fc1rMRMvMOjNWuyfszOwVYiqXARHBQk50mZIl+N+/8levnS+9Riy3zeGmqbHutBwtTFn7N4zu+DyV95pcUk+8kWp1uf9JJCnVD4kl1DmKpe11Cvhv3hUvTiYs2QN/kZRHzn2wjI3ODkd1gx/i2uDqnCZj61bf24XRg6QUuNMx4Gt5aNe7G/qQTIsuTGnGzRMkd5aUYS74AUOPhtMisYilCQ1r7LoQ3e+opANXy91o0j0zq6hrisKoTWom6CNcSVIGDEupRImE9jPRlYtb8rCr/T7jBtFac10sLtR4G23/ShCBPCZqk3KhOJbp5xboiT3h5wbk0T4RkrL+8= ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968 2024-06-18T02:30:35.549 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:34.787+0000 7f172aad6640 1 Processor -- start 2024-06-18T02:30:35.549 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:34.787+0000 7f172aad6640 1 -- start start 2024-06-18T02:30:35.549 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:34.787+0000 7f172aad6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1724101d00 0x7f1724102110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:35.549 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:34.787+0000 7f172aad6640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1724102650 con 0x7f1724101d00 2024-06-18T02:30:35.549 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:34.787+0000 7f1728872640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1724101d00 0x7f1724102110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:30:35.549 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:34.787+0000 7f1728872640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1724101d00 0x7f1724102110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57710/0 (socket says 172.21.15.1:57710) 2024-06-18T02:30:35.550 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:34.787+0000 7f1728872640 1 -- 172.21.15.1:0/3921268791 learned_addr learned my addr 172.21.15.1:0/3921268791 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:30:35.550 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:34.787+0000 7f1728872640 1 -- 172.21.15.1:0/3921268791 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1724102790 con 0x7f1724101d00 2024-06-18T02:30:35.550 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:34.788+0000 7f1728872640 1 --2- 172.21.15.1:0/3921268791 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1724101d00 0x7f1724102110 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1718009840 tx=0x7f17180091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bb54f5bea06bb52 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:30:35.550 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:34.788+0000 7f17237fe640 1 -- 172.21.15.1:0/3921268791 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1718032440 con 0x7f1724101d00 2024-06-18T02:30:35.550 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:34.789+0000 7f17237fe640 1 -- 172.21.15.1:0/3921268791 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f1718032a30 con 0x7f1724101d00 2024-06-18T02:30:35.550 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:34.789+0000 7f17237fe640 1 -- 172.21.15.1:0/3921268791 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1718031ad0 con 0x7f1724101d00 2024-06-18T02:30:35.550 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:34.789+0000 7f172aad6640 1 -- 172.21.15.1:0/3921268791 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1724101d00 msgr2=0x7f1724102110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:30:35.551 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:34.789+0000 7f172aad6640 1 --2- 172.21.15.1:0/3921268791 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1724101d00 0x7f1724102110 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1718009840 tx=0x7f17180091d0 comp rx=0 tx=0).stop 2024-06-18T02:30:35.551 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:34.789+0000 7f172aad6640 1 -- 172.21.15.1:0/3921268791 shutdown_connections 2024-06-18T02:30:35.551 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:34.789+0000 7f172aad6640 1 --2- 172.21.15.1:0/3921268791 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1724101d00 0x7f1724102110 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:35.551 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:34.789+0000 7f172aad6640 1 -- 172.21.15.1:0/3921268791 >> 172.21.15.1:0/3921268791 conn(0x7f17240fd500 msgr2=0x7f17240ff910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:30:35.551 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:34.789+0000 7f172aad6640 1 -- 172.21.15.1:0/3921268791 shutdown_connections 2024-06-18T02:30:35.551 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:34.789+0000 7f172aad6640 1 -- 172.21.15.1:0/3921268791 wait complete. 2024-06-18T02:30:35.551 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:34.790+0000 7f172aad6640 1 Processor -- start 2024-06-18T02:30:35.551 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:34.790+0000 7f172aad6640 1 -- start start 2024-06-18T02:30:35.551 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:34.790+0000 7f172aad6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1724101d00 0x7f1724110ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:35.551 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:34.790+0000 7f172aad6640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17241113e0 con 0x7f1724101d00 2024-06-18T02:30:35.552 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:34.790+0000 7f1728872640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1724101d00 0x7f1724110ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:30:35.552 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:34.790+0000 7f1728872640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1724101d00 0x7f1724110ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57716/0 (socket says 172.21.15.1:57716) 2024-06-18T02:30:35.552 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:34.790+0000 7f1728872640 1 -- 172.21.15.1:0/412404559 learned_addr learned my addr 172.21.15.1:0/412404559 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:30:35.552 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:34.790+0000 7f1728872640 1 -- 172.21.15.1:0/412404559 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1718009450 con 0x7f1724101d00 2024-06-18T02:30:35.552 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:34.791+0000 7f1728872640 1 --2- 172.21.15.1:0/412404559 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1724101d00 0x7f1724110ea0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1718009420 tx=0x7f1718008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:30:35.552 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:34.791+0000 7f17217fa640 1 -- 172.21.15.1:0/412404559 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f171803d5b0 con 0x7f1724101d00 2024-06-18T02:30:35.552 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:34.791+0000 7f17217fa640 1 -- 172.21.15.1:0/412404559 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f171803dba0 con 0x7f1724101d00 2024-06-18T02:30:35.552 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:34.791+0000 7f172aad6640 1 -- 172.21.15.1:0/412404559 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17241115e0 con 0x7f1724101d00 2024-06-18T02:30:35.552 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:34.791+0000 7f17217fa640 1 -- 172.21.15.1:0/412404559 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f171803bb40 con 0x7f1724101d00 2024-06-18T02:30:35.552 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:34.791+0000 7f172aad6640 1 -- 172.21.15.1:0/412404559 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1724111fe0 con 0x7f1724101d00 2024-06-18T02:30:35.552 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:34.792+0000 7f17217fa640 1 -- 172.21.15.1:0/412404559 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f1718039070 con 0x7f1724101d00 2024-06-18T02:30:35.552 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:34.792+0000 7f17217fa640 1 --2- 172.21.15.1:0/412404559 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f1714034860 0x7f1714036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:35.552 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:34.792+0000 7f172aad6640 1 -- 172.21.15.1:0/412404559 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f172405ead0 con 0x7f1724101d00 2024-06-18T02:30:35.552 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:34.792+0000 7f17217fa640 1 -- 172.21.15.1:0/412404559 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f1718067a90 con 0x7f1724101d00 2024-06-18T02:30:35.553 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:34.792+0000 7f1723fff640 1 --2- 172.21.15.1:0/412404559 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f1714034860 0x7f1714036d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:30:35.553 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:34.792+0000 7f1723fff640 1 --2- 172.21.15.1:0/412404559 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f1714034860 0x7f1714036d10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1710006fd0 tx=0x7f1710006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:30:35.553 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:34.796+0000 7f17217fa640 1 -- 172.21.15.1:0/412404559 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f1718041020 con 0x7f1724101d00 2024-06-18T02:30:35.553 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:34.973+0000 7f172aad6640 1 -- 172.21.15.1:0/412404559 --> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f1724112220 con 0x7f1714034860 2024-06-18T02:30:35.553 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:34.973+0000 7f17217fa640 1 -- 172.21.15.1:0/412404559 <== mgr.14118 v2:172.21.15.1:6800/4268462134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f1724112220 con 0x7f1714034860 2024-06-18T02:30:35.553 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:34.975+0000 7f172aad6640 1 -- 172.21.15.1:0/412404559 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f1714034860 msgr2=0x7f1714036d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:30:35.553 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:34.975+0000 7f172aad6640 1 --2- 172.21.15.1:0/412404559 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f1714034860 0x7f1714036d10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1710006fd0 tx=0x7f1710006db0 comp rx=0 tx=0).stop 2024-06-18T02:30:35.553 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:34.975+0000 7f172aad6640 1 -- 172.21.15.1:0/412404559 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1724101d00 msgr2=0x7f1724110ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:30:35.553 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:34.975+0000 7f172aad6640 1 --2- 172.21.15.1:0/412404559 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1724101d00 0x7f1724110ea0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1718009420 tx=0x7f1718008040 comp rx=0 tx=0).stop 2024-06-18T02:30:35.553 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:34.976+0000 7f172aad6640 1 -- 172.21.15.1:0/412404559 shutdown_connections 2024-06-18T02:30:35.554 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:34.976+0000 7f172aad6640 1 --2- 172.21.15.1:0/412404559 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f1714034860 0x7f1714036d10 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:35.554 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:34.976+0000 7f172aad6640 1 --2- 172.21.15.1:0/412404559 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1724101d00 0x7f1724110ea0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:35.554 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:34.976+0000 7f172aad6640 1 -- 172.21.15.1:0/412404559 >> 172.21.15.1:0/412404559 conn(0x7f17240fd500 msgr2=0x7f17240fe200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:30:35.554 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:34.976+0000 7f172aad6640 1 -- 172.21.15.1:0/412404559 shutdown_connections 2024-06-18T02:30:35.554 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:34.976+0000 7f172aad6640 1 -- 172.21.15.1:0/412404559 wait complete. 2024-06-18T02:30:35.554 INFO:teuthology.orchestra.run.smithi001.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-06-18T02:30:35.554 INFO:teuthology.orchestra.run.smithi001.stdout:Adding key to root@localhost authorized_keys... 2024-06-18T02:30:35.554 INFO:teuthology.orchestra.run.smithi001.stdout:Adding host smithi001... 2024-06-18T02:30:36.127 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:35 smithi001 ceph-mon[29099]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-06-18T02:30:37.127 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:36 smithi001 ceph-mon[29099]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi001", "addr": "172.21.15.1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-18T02:30:38.098 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:37 smithi001 ceph-mon[29099]: Deploying cephadm binary to smithi001 2024-06-18T02:30:40.102 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout Added host 'smithi001' with addr '172.21.15.1' 2024-06-18T02:30:40.102 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:35.954+0000 7f348b659640 1 Processor -- start 2024-06-18T02:30:40.102 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:35.954+0000 7f348b659640 1 -- start start 2024-06-18T02:30:40.103 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:35.954+0000 7f348b659640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3484075bc0 0x7f3484106d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:40.103 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:35.954+0000 7f348b659640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3484102760 con 0x7f3484075bc0 2024-06-18T02:30:40.103 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:35.954+0000 7f34893f5640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3484075bc0 0x7f3484106d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:30:40.103 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:35.954+0000 7f34893f5640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3484075bc0 0x7f3484106d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57726/0 (socket says 172.21.15.1:57726) 2024-06-18T02:30:40.103 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:35.954+0000 7f34893f5640 1 -- 172.21.15.1:0/2602151672 learned_addr learned my addr 172.21.15.1:0/2602151672 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:30:40.103 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:35.955+0000 7f34893f5640 1 -- 172.21.15.1:0/2602151672 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34841072b0 con 0x7f3484075bc0 2024-06-18T02:30:40.103 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:35.955+0000 7f34893f5640 1 --2- 172.21.15.1:0/2602151672 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3484075bc0 0x7f3484106d70 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3480009840 tx=0x7f34800091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58cd5c89c1b8bf94 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:30:40.103 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:35.956+0000 7f3477fff640 1 -- 172.21.15.1:0/2602151672 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3480032440 con 0x7f3484075bc0 2024-06-18T02:30:40.103 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:35.956+0000 7f3477fff640 1 -- 172.21.15.1:0/2602151672 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f3480032a30 con 0x7f3484075bc0 2024-06-18T02:30:40.103 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:35.956+0000 7f3477fff640 1 -- 172.21.15.1:0/2602151672 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3480031ad0 con 0x7f3484075bc0 2024-06-18T02:30:40.103 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:35.956+0000 7f348b659640 1 -- 172.21.15.1:0/2602151672 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3484075bc0 msgr2=0x7f3484106d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:30:40.103 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:35.956+0000 7f348b659640 1 --2- 172.21.15.1:0/2602151672 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3484075bc0 0x7f3484106d70 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3480009840 tx=0x7f34800091d0 comp rx=0 tx=0).stop 2024-06-18T02:30:40.104 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:35.956+0000 7f348b659640 1 -- 172.21.15.1:0/2602151672 shutdown_connections 2024-06-18T02:30:40.104 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:35.956+0000 7f348b659640 1 --2- 172.21.15.1:0/2602151672 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3484075bc0 0x7f3484106d70 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:40.104 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:35.956+0000 7f348b659640 1 -- 172.21.15.1:0/2602151672 >> 172.21.15.1:0/2602151672 conn(0x7f34840fc480 msgr2=0x7f34840fe890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:30:40.104 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:35.956+0000 7f348b659640 1 -- 172.21.15.1:0/2602151672 shutdown_connections 2024-06-18T02:30:40.104 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:35.956+0000 7f348b659640 1 -- 172.21.15.1:0/2602151672 wait complete. 2024-06-18T02:30:40.104 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:35.957+0000 7f348b659640 1 Processor -- start 2024-06-18T02:30:40.104 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:35.957+0000 7f348b659640 1 -- start start 2024-06-18T02:30:40.104 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:35.957+0000 7f348b659640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3484075bc0 0x7f3484112d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:40.104 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:35.957+0000 7f348b659640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3484113250 con 0x7f3484075bc0 2024-06-18T02:30:40.104 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:35.957+0000 7f34893f5640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3484075bc0 0x7f3484112d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:30:40.104 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:35.957+0000 7f34893f5640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3484075bc0 0x7f3484112d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57730/0 (socket says 172.21.15.1:57730) 2024-06-18T02:30:40.105 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:35.957+0000 7f34893f5640 1 -- 172.21.15.1:0/4057727117 learned_addr learned my addr 172.21.15.1:0/4057727117 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:30:40.105 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:35.958+0000 7f34893f5640 1 -- 172.21.15.1:0/4057727117 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3480009450 con 0x7f3484075bc0 2024-06-18T02:30:40.105 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:35.958+0000 7f34893f5640 1 --2- 172.21.15.1:0/4057727117 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3484075bc0 0x7f3484112d10 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3480031b90 tx=0x7f3480008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:30:40.105 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:35.958+0000 7f3475ffb640 1 -- 172.21.15.1:0/4057727117 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34800328b0 con 0x7f3484075bc0 2024-06-18T02:30:40.105 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:35.958+0000 7f3475ffb640 1 -- 172.21.15.1:0/4057727117 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f348003d8e0 con 0x7f3484075bc0 2024-06-18T02:30:40.105 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:35.958+0000 7f348b659640 1 -- 172.21.15.1:0/4057727117 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3484113450 con 0x7f3484075bc0 2024-06-18T02:30:40.105 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:35.958+0000 7f3475ffb640 1 -- 172.21.15.1:0/4057727117 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f348003ba20 con 0x7f3484075bc0 2024-06-18T02:30:40.105 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:35.958+0000 7f348b659640 1 -- 172.21.15.1:0/4057727117 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34841138c0 con 0x7f3484075bc0 2024-06-18T02:30:40.105 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:35.959+0000 7f3475ffb640 1 -- 172.21.15.1:0/4057727117 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f3480039070 con 0x7f3484075bc0 2024-06-18T02:30:40.105 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:35.959+0000 7f3475ffb640 1 --2- 172.21.15.1:0/4057727117 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f3470034810 0x7f3470036cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:40.105 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:35.959+0000 7f3475ffb640 1 -- 172.21.15.1:0/4057727117 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f3480067880 con 0x7f3484075bc0 2024-06-18T02:30:40.106 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:35.959+0000 7f348b659640 1 -- 172.21.15.1:0/4057727117 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f348405ead0 con 0x7f3484075bc0 2024-06-18T02:30:40.106 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:35.959+0000 7f3488bf4640 1 --2- 172.21.15.1:0/4057727117 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f3470034810 0x7f3470036cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:30:40.106 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:35.960+0000 7f3488bf4640 1 --2- 172.21.15.1:0/4057727117 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f3470034810 0x7f3470036cc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f3478006fd0 tx=0x7f3478006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:30:40.106 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:35.964+0000 7f3475ffb640 1 -- 172.21.15.1:0/4057727117 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f3480041070 con 0x7f3484075bc0 2024-06-18T02:30:40.106 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:36.143+0000 7f348b659640 1 -- 172.21.15.1:0/4057727117 --> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi001", "addr": "172.21.15.1", "target": ["mon-mgr", ""]}) v1 -- 0x7f34840fd700 con 0x7f3470034810 2024-06-18T02:30:40.106 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:39.557+0000 7f3475ffb640 1 -- 172.21.15.1:0/4057727117 <== mgr.14118 v2:172.21.15.1:6800/4268462134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f34840fd700 con 0x7f3470034810 2024-06-18T02:30:40.106 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:39.560+0000 7f348b659640 1 -- 172.21.15.1:0/4057727117 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f3470034810 msgr2=0x7f3470036cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:30:40.106 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:39.560+0000 7f348b659640 1 --2- 172.21.15.1:0/4057727117 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f3470034810 0x7f3470036cc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f3478006fd0 tx=0x7f3478006db0 comp rx=0 tx=0).stop 2024-06-18T02:30:40.106 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:39.560+0000 7f348b659640 1 -- 172.21.15.1:0/4057727117 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3484075bc0 msgr2=0x7f3484112d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:30:40.106 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:39.560+0000 7f348b659640 1 --2- 172.21.15.1:0/4057727117 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3484075bc0 0x7f3484112d10 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3480031b90 tx=0x7f3480008040 comp rx=0 tx=0).stop 2024-06-18T02:30:40.107 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:39.560+0000 7f348b659640 1 -- 172.21.15.1:0/4057727117 shutdown_connections 2024-06-18T02:30:40.107 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:39.560+0000 7f348b659640 1 --2- 172.21.15.1:0/4057727117 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f3470034810 0x7f3470036cc0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:40.107 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:39.560+0000 7f348b659640 1 --2- 172.21.15.1:0/4057727117 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3484075bc0 0x7f3484112d10 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:40.107 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:39.560+0000 7f348b659640 1 -- 172.21.15.1:0/4057727117 >> 172.21.15.1:0/4057727117 conn(0x7f34840fc480 msgr2=0x7f34840fcfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:30:40.107 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:39.560+0000 7f348b659640 1 -- 172.21.15.1:0/4057727117 shutdown_connections 2024-06-18T02:30:40.107 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:39.560+0000 7f348b659640 1 -- 172.21.15.1:0/4057727117 wait complete. 2024-06-18T02:30:40.107 INFO:teuthology.orchestra.run.smithi001.stdout:Deploying mon service with default placement... 2024-06-18T02:30:40.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:40 smithi001 ceph-mon[29099]: from='mgr.14118 172.21.15.1:0/2084661558' entity='mgr.smithi001.teoawz' 2024-06-18T02:30:40.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:40 smithi001 ceph-mon[29099]: Added host smithi001 2024-06-18T02:30:40.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:40 smithi001 ceph-mon[29099]: from='mgr.14118 172.21.15.1:0/2084661558' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:30:41.538 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-06-18T02:30:41.538 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:40.512+0000 7f5b03fba640 1 Processor -- start 2024-06-18T02:30:41.538 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:40.512+0000 7f5b03fba640 1 -- start start 2024-06-18T02:30:41.538 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:40.512+0000 7f5b03fba640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5afc075b10 0x7f5afc104a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:41.538 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:40.512+0000 7f5b03fba640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5afc075f20 con 0x7f5afc075b10 2024-06-18T02:30:41.539 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:40.513+0000 7f5b01d56640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5afc075b10 0x7f5afc104a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:30:41.539 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:40.513+0000 7f5b01d56640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5afc075b10 0x7f5afc104a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57742/0 (socket says 172.21.15.1:57742) 2024-06-18T02:30:41.539 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:40.513+0000 7f5b01d56640 1 -- 172.21.15.1:0/2632886450 learned_addr learned my addr 172.21.15.1:0/2632886450 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:30:41.539 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:40.513+0000 7f5b01d56640 1 -- 172.21.15.1:0/2632886450 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5afc104f70 con 0x7f5afc075b10 2024-06-18T02:30:41.539 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:40.514+0000 7f5b01d56640 1 --2- 172.21.15.1:0/2632886450 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5afc075b10 0x7f5afc104a30 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f5af8009840 tx=0x7f5af80091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7f96ed9f1d6f6a1 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:30:41.539 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:40.514+0000 7f5b00d54640 1 -- 172.21.15.1:0/2632886450 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5af8032440 con 0x7f5afc075b10 2024-06-18T02:30:41.539 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:40.514+0000 7f5b00d54640 1 -- 172.21.15.1:0/2632886450 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f5af8032a30 con 0x7f5afc075b10 2024-06-18T02:30:41.539 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:40.514+0000 7f5b00d54640 1 -- 172.21.15.1:0/2632886450 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5af8031ad0 con 0x7f5afc075b10 2024-06-18T02:30:41.539 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:40.515+0000 7f5b03fba640 1 -- 172.21.15.1:0/2632886450 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5afc075b10 msgr2=0x7f5afc104a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:30:41.540 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:40.515+0000 7f5b03fba640 1 --2- 172.21.15.1:0/2632886450 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5afc075b10 0x7f5afc104a30 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f5af8009840 tx=0x7f5af80091d0 comp rx=0 tx=0).stop 2024-06-18T02:30:41.540 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:40.515+0000 7f5b03fba640 1 -- 172.21.15.1:0/2632886450 shutdown_connections 2024-06-18T02:30:41.540 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:40.515+0000 7f5b03fba640 1 --2- 172.21.15.1:0/2632886450 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5afc075b10 0x7f5afc104a30 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:41.540 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:40.515+0000 7f5b03fba640 1 -- 172.21.15.1:0/2632886450 >> 172.21.15.1:0/2632886450 conn(0x7f5afc0fc460 msgr2=0x7f5afc0fe870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:30:41.540 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:40.515+0000 7f5b03fba640 1 -- 172.21.15.1:0/2632886450 shutdown_connections 2024-06-18T02:30:41.540 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:40.515+0000 7f5b03fba640 1 -- 172.21.15.1:0/2632886450 wait complete. 2024-06-18T02:30:41.540 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:40.515+0000 7f5b03fba640 1 Processor -- start 2024-06-18T02:30:41.540 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:40.515+0000 7f5b03fba640 1 -- start start 2024-06-18T02:30:41.540 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:40.515+0000 7f5b03fba640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5afc075b10 0x7f5afc110b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:41.540 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:40.515+0000 7f5b03fba640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5afc111040 con 0x7f5afc075b10 2024-06-18T02:30:41.540 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:40.516+0000 7f5b01d56640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5afc075b10 0x7f5afc110b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:30:41.540 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:40.516+0000 7f5b01d56640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5afc075b10 0x7f5afc110b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57752/0 (socket says 172.21.15.1:57752) 2024-06-18T02:30:41.541 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:40.516+0000 7f5b01d56640 1 -- 172.21.15.1:0/189775872 learned_addr learned my addr 172.21.15.1:0/189775872 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:30:41.541 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:40.516+0000 7f5b01d56640 1 -- 172.21.15.1:0/189775872 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5af8009450 con 0x7f5afc075b10 2024-06-18T02:30:41.541 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:40.516+0000 7f5b01d56640 1 --2- 172.21.15.1:0/189775872 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5afc075b10 0x7f5afc110b00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f5af8006f00 tx=0x7f5af8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:30:41.541 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:40.516+0000 7f5af27fc640 1 -- 172.21.15.1:0/189775872 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5af803d450 con 0x7f5afc075b10 2024-06-18T02:30:41.541 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:40.516+0000 7f5af27fc640 1 -- 172.21.15.1:0/189775872 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f5af803da40 con 0x7f5afc075b10 2024-06-18T02:30:41.541 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:40.516+0000 7f5af27fc640 1 -- 172.21.15.1:0/189775872 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5af803bb10 con 0x7f5afc075b10 2024-06-18T02:30:41.541 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:40.516+0000 7f5b03fba640 1 -- 172.21.15.1:0/189775872 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5afc111240 con 0x7f5afc075b10 2024-06-18T02:30:41.541 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:40.516+0000 7f5b03fba640 1 -- 172.21.15.1:0/189775872 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5afc1116b0 con 0x7f5afc075b10 2024-06-18T02:30:41.541 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:40.517+0000 7f5af27fc640 1 -- 172.21.15.1:0/189775872 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f5af8039070 con 0x7f5afc075b10 2024-06-18T02:30:41.541 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:40.517+0000 7f5af27fc640 1 --2- 172.21.15.1:0/189775872 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f5ae8034860 0x7f5ae8036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:41.541 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:40.517+0000 7f5af27fc640 1 -- 172.21.15.1:0/189775872 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f5af80675b0 con 0x7f5afc075b10 2024-06-18T02:30:41.541 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:40.517+0000 7f5b03fba640 1 -- 172.21.15.1:0/189775872 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5afc05ead0 con 0x7f5afc075b10 2024-06-18T02:30:41.542 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:40.517+0000 7f5b01555640 1 --2- 172.21.15.1:0/189775872 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f5ae8034860 0x7f5ae8036d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:30:41.542 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:40.518+0000 7f5b01555640 1 --2- 172.21.15.1:0/189775872 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f5ae8034860 0x7f5ae8036d10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f5aec006fd0 tx=0x7f5aec006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:30:41.542 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:40.522+0000 7f5af27fc640 1 -- 172.21.15.1:0/189775872 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f5af8041080 con 0x7f5afc075b10 2024-06-18T02:30:41.542 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:40.706+0000 7f5b03fba640 1 -- 172.21.15.1:0/189775872 --> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f5afc0fd700 con 0x7f5ae8034860 2024-06-18T02:30:41.542 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:40.711+0000 7f5af27fc640 1 -- 172.21.15.1:0/189775872 <== mgr.14118 v2:172.21.15.1:6800/4268462134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f5afc0fd700 con 0x7f5ae8034860 2024-06-18T02:30:41.542 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:40.712+0000 7f5b03fba640 1 -- 172.21.15.1:0/189775872 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f5ae8034860 msgr2=0x7f5ae8036d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:30:41.542 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:40.713+0000 7f5b03fba640 1 --2- 172.21.15.1:0/189775872 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f5ae8034860 0x7f5ae8036d10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f5aec006fd0 tx=0x7f5aec006db0 comp rx=0 tx=0).stop 2024-06-18T02:30:41.542 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:40.713+0000 7f5b03fba640 1 -- 172.21.15.1:0/189775872 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5afc075b10 msgr2=0x7f5afc110b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:30:41.543 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:40.713+0000 7f5b03fba640 1 --2- 172.21.15.1:0/189775872 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5afc075b10 0x7f5afc110b00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f5af8006f00 tx=0x7f5af8008040 comp rx=0 tx=0).stop 2024-06-18T02:30:41.543 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:40.713+0000 7f5b03fba640 1 -- 172.21.15.1:0/189775872 shutdown_connections 2024-06-18T02:30:41.543 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:40.713+0000 7f5b03fba640 1 --2- 172.21.15.1:0/189775872 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f5ae8034860 0x7f5ae8036d10 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:41.543 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:40.713+0000 7f5b03fba640 1 --2- 172.21.15.1:0/189775872 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5afc075b10 0x7f5afc110b00 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:41.543 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:40.713+0000 7f5b03fba640 1 -- 172.21.15.1:0/189775872 >> 172.21.15.1:0/189775872 conn(0x7f5afc0fc460 msgr2=0x7f5afc0fcfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:30:41.543 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:40.713+0000 7f5b03fba640 1 -- 172.21.15.1:0/189775872 shutdown_connections 2024-06-18T02:30:41.543 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:40.713+0000 7f5b03fba640 1 -- 172.21.15.1:0/189775872 wait complete. 2024-06-18T02:30:41.543 INFO:teuthology.orchestra.run.smithi001.stdout:Deploying mgr service with default placement... 2024-06-18T02:30:41.825 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:41 smithi001 ceph-mon[29099]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-06-18T02:30:41.825 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:41 smithi001 ceph-mon[29099]: Saving service mon spec with placement count:5 2024-06-18T02:30:41.825 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:41 smithi001 ceph-mon[29099]: from='mgr.14118 172.21.15.1:0/2084661558' entity='mgr.smithi001.teoawz' 2024-06-18T02:30:43.241 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:43 smithi001 ceph-mon[29099]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-06-18T02:30:43.241 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:43 smithi001 ceph-mon[29099]: Saving service mgr spec with placement count:2 2024-06-18T02:30:43.242 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:43 smithi001 ceph-mon[29099]: from='mgr.14118 172.21.15.1:0/2084661558' entity='mgr.smithi001.teoawz' 2024-06-18T02:30:43.242 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:43 smithi001 ceph-mon[29099]: from='mgr.14118 172.21.15.1:0/2084661558' entity='mgr.smithi001.teoawz' 2024-06-18T02:30:43.248 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-06-18T02:30:43.248 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:41.909+0000 7f7acb436640 1 Processor -- start 2024-06-18T02:30:43.248 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:41.909+0000 7f7acb436640 1 -- start start 2024-06-18T02:30:43.248 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:41.909+0000 7f7acb436640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7ac4100ca0 0x7f7ac41010b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:43.249 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:41.909+0000 7f7acb436640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac41015f0 con 0x7f7ac4100ca0 2024-06-18T02:30:43.249 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:41.910+0000 7f7ac91d2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7ac4100ca0 0x7f7ac41010b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:30:43.249 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:41.910+0000 7f7ac91d2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7ac4100ca0 0x7f7ac41010b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57768/0 (socket says 172.21.15.1:57768) 2024-06-18T02:30:43.249 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:41.910+0000 7f7ac91d2640 1 -- 172.21.15.1:0/2386646127 learned_addr learned my addr 172.21.15.1:0/2386646127 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:30:43.249 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:41.910+0000 7f7ac91d2640 1 -- 172.21.15.1:0/2386646127 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ac4101730 con 0x7f7ac4100ca0 2024-06-18T02:30:43.249 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:41.911+0000 7f7ac91d2640 1 --2- 172.21.15.1:0/2386646127 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7ac4100ca0 0x7f7ac41010b0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7ac0009840 tx=0x7f7ac00091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8db9df0e94e1429c server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:30:43.249 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:41.911+0000 7f7abbfff640 1 -- 172.21.15.1:0/2386646127 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ac0032440 con 0x7f7ac4100ca0 2024-06-18T02:30:43.249 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:41.911+0000 7f7abbfff640 1 -- 172.21.15.1:0/2386646127 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f7ac0032a30 con 0x7f7ac4100ca0 2024-06-18T02:30:43.249 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:41.911+0000 7f7abbfff640 1 -- 172.21.15.1:0/2386646127 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ac0031ad0 con 0x7f7ac4100ca0 2024-06-18T02:30:43.249 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:41.911+0000 7f7acb436640 1 -- 172.21.15.1:0/2386646127 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7ac4100ca0 msgr2=0x7f7ac41010b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:30:43.249 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:41.911+0000 7f7acb436640 1 --2- 172.21.15.1:0/2386646127 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7ac4100ca0 0x7f7ac41010b0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7ac0009840 tx=0x7f7ac00091d0 comp rx=0 tx=0).stop 2024-06-18T02:30:43.249 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:41.911+0000 7f7acb436640 1 -- 172.21.15.1:0/2386646127 shutdown_connections 2024-06-18T02:30:43.249 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:41.911+0000 7f7acb436640 1 --2- 172.21.15.1:0/2386646127 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7ac4100ca0 0x7f7ac41010b0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:43.249 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:41.911+0000 7f7acb436640 1 -- 172.21.15.1:0/2386646127 >> 172.21.15.1:0/2386646127 conn(0x7f7ac40fc4a0 msgr2=0x7f7ac40fe8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:30:43.250 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:41.912+0000 7f7acb436640 1 -- 172.21.15.1:0/2386646127 shutdown_connections 2024-06-18T02:30:43.250 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:41.912+0000 7f7acb436640 1 -- 172.21.15.1:0/2386646127 wait complete. 2024-06-18T02:30:43.250 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:41.912+0000 7f7acb436640 1 Processor -- start 2024-06-18T02:30:43.250 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:41.912+0000 7f7acb436640 1 -- start start 2024-06-18T02:30:43.250 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:41.912+0000 7f7acb436640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7ac4100ca0 0x7f7ac4106ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:43.250 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:41.912+0000 7f7acb436640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac4107400 con 0x7f7ac4100ca0 2024-06-18T02:30:43.250 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:41.913+0000 7f7ac91d2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7ac4100ca0 0x7f7ac4106ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:30:43.250 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:41.913+0000 7f7ac91d2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7ac4100ca0 0x7f7ac4106ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57782/0 (socket says 172.21.15.1:57782) 2024-06-18T02:30:43.250 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:41.913+0000 7f7ac91d2640 1 -- 172.21.15.1:0/2676092217 learned_addr learned my addr 172.21.15.1:0/2676092217 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:30:43.250 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:41.913+0000 7f7ac91d2640 1 -- 172.21.15.1:0/2676092217 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ac0009450 con 0x7f7ac4100ca0 2024-06-18T02:30:43.250 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:41.913+0000 7f7ac91d2640 1 --2- 172.21.15.1:0/2676092217 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7ac4100ca0 0x7f7ac4106ec0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7ac0031670 tx=0x7f7ac0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:30:43.250 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:41.913+0000 7f7ab9ffb640 1 -- 172.21.15.1:0/2676092217 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ac003d450 con 0x7f7ac4100ca0 2024-06-18T02:30:43.250 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:41.913+0000 7f7ab9ffb640 1 -- 172.21.15.1:0/2676092217 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f7ac003da40 con 0x7f7ac4100ca0 2024-06-18T02:30:43.251 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:41.913+0000 7f7ab9ffb640 1 -- 172.21.15.1:0/2676092217 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ac003ba00 con 0x7f7ac4100ca0 2024-06-18T02:30:43.251 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:41.913+0000 7f7acb436640 1 -- 172.21.15.1:0/2676092217 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ac4105490 con 0x7f7ac4100ca0 2024-06-18T02:30:43.251 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:41.914+0000 7f7acb436640 1 -- 172.21.15.1:0/2676092217 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ac4105900 con 0x7f7ac4100ca0 2024-06-18T02:30:43.251 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:41.914+0000 7f7ab9ffb640 1 -- 172.21.15.1:0/2676092217 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f7ac0039070 con 0x7f7ac4100ca0 2024-06-18T02:30:43.251 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:41.915+0000 7f7ab9ffb640 1 --2- 172.21.15.1:0/2676092217 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f7ab0034860 0x7f7ab0036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:43.251 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:41.915+0000 7f7acb436640 1 -- 172.21.15.1:0/2676092217 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ac405ead0 con 0x7f7ac4100ca0 2024-06-18T02:30:43.251 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:41.915+0000 7f7ab9ffb640 1 -- 172.21.15.1:0/2676092217 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f7ac0067790 con 0x7f7ac4100ca0 2024-06-18T02:30:43.251 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:41.915+0000 7f7ac89d1640 1 --2- 172.21.15.1:0/2676092217 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f7ab0034860 0x7f7ab0036d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:30:43.251 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:41.915+0000 7f7ac89d1640 1 --2- 172.21.15.1:0/2676092217 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f7ab0034860 0x7f7ab0036d10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7ab4006fd0 tx=0x7f7ab4006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:30:43.251 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:41.920+0000 7f7ab9ffb640 1 -- 172.21.15.1:0/2676092217 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f7ac0037030 con 0x7f7ac4100ca0 2024-06-18T02:30:43.251 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:42.106+0000 7f7acb436640 1 -- 172.21.15.1:0/2676092217 --> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f7ac410a970 con 0x7f7ab0034860 2024-06-18T02:30:43.251 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:42.109+0000 7f7ab9ffb640 1 -- 172.21.15.1:0/2676092217 <== mgr.14118 v2:172.21.15.1:6800/4268462134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f7ac410a970 con 0x7f7ab0034860 2024-06-18T02:30:43.251 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:42.112+0000 7f7acb436640 1 -- 172.21.15.1:0/2676092217 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f7ab0034860 msgr2=0x7f7ab0036d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:30:43.251 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:42.112+0000 7f7acb436640 1 --2- 172.21.15.1:0/2676092217 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f7ab0034860 0x7f7ab0036d10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7ab4006fd0 tx=0x7f7ab4006db0 comp rx=0 tx=0).stop 2024-06-18T02:30:43.252 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:42.112+0000 7f7acb436640 1 -- 172.21.15.1:0/2676092217 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7ac4100ca0 msgr2=0x7f7ac4106ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:30:43.252 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:42.112+0000 7f7acb436640 1 --2- 172.21.15.1:0/2676092217 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7ac4100ca0 0x7f7ac4106ec0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7ac0031670 tx=0x7f7ac0008040 comp rx=0 tx=0).stop 2024-06-18T02:30:43.252 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:42.112+0000 7f7acb436640 1 -- 172.21.15.1:0/2676092217 shutdown_connections 2024-06-18T02:30:43.252 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:42.112+0000 7f7acb436640 1 --2- 172.21.15.1:0/2676092217 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f7ab0034860 0x7f7ab0036d10 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:43.252 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:42.112+0000 7f7acb436640 1 --2- 172.21.15.1:0/2676092217 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7ac4100ca0 0x7f7ac4106ec0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:43.252 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:42.112+0000 7f7acb436640 1 -- 172.21.15.1:0/2676092217 >> 172.21.15.1:0/2676092217 conn(0x7f7ac40fc4a0 msgr2=0x7f7ac40fd370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:30:43.252 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:42.112+0000 7f7acb436640 1 -- 172.21.15.1:0/2676092217 shutdown_connections 2024-06-18T02:30:43.252 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:42.112+0000 7f7acb436640 1 -- 172.21.15.1:0/2676092217 wait complete. 2024-06-18T02:30:43.252 INFO:teuthology.orchestra.run.smithi001.stdout:Deploying crash service with default placement... 2024-06-18T02:30:44.534 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:44 smithi001 ceph-mon[29099]: from='mgr.14118 172.21.15.1:0/2084661558' entity='mgr.smithi001.teoawz' 2024-06-18T02:30:44.535 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:44 smithi001 ceph-mon[29099]: from='mgr.14118 172.21.15.1:0/2084661558' entity='mgr.smithi001.teoawz' 2024-06-18T02:30:45.095 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout Scheduled crash update... 2024-06-18T02:30:45.101 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:43.689+0000 7f0224a02640 1 Processor -- start 2024-06-18T02:30:45.101 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:43.689+0000 7f0224a02640 1 -- start start 2024-06-18T02:30:45.101 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:43.689+0000 7f0224a02640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0220100010 0x7f0220100440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:45.101 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:43.689+0000 7f0224a02640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0220102e90 con 0x7f0220100010 2024-06-18T02:30:45.101 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:43.689+0000 7f021ed9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0220100010 0x7f0220100440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:30:45.101 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:43.690+0000 7f021ed9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0220100010 0x7f0220100440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57796/0 (socket says 172.21.15.1:57796) 2024-06-18T02:30:45.101 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:43.690+0000 7f021ed9d640 1 -- 172.21.15.1:0/2931518352 learned_addr learned my addr 172.21.15.1:0/2931518352 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:30:45.101 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:43.690+0000 7f021ed9d640 1 -- 172.21.15.1:0/2931518352 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0220100980 con 0x7f0220100010 2024-06-18T02:30:45.101 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:43.690+0000 7f021ed9d640 1 --2- 172.21.15.1:0/2931518352 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0220100010 0x7f0220100440 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0210009840 tx=0x7f02100091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=235f31b56ff4a4e6 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:30:45.102 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:43.691+0000 7f021dd9b640 1 -- 172.21.15.1:0/2931518352 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0210032440 con 0x7f0220100010 2024-06-18T02:30:45.102 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:43.691+0000 7f021dd9b640 1 -- 172.21.15.1:0/2931518352 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0210032a70 con 0x7f0220100010 2024-06-18T02:30:45.102 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:43.691+0000 7f021dd9b640 1 -- 172.21.15.1:0/2931518352 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0210031ad0 con 0x7f0220100010 2024-06-18T02:30:45.102 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:43.691+0000 7f0224a02640 1 -- 172.21.15.1:0/2931518352 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0220100010 msgr2=0x7f0220100440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:30:45.102 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:43.691+0000 7f0224a02640 1 --2- 172.21.15.1:0/2931518352 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0220100010 0x7f0220100440 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0210009840 tx=0x7f02100091d0 comp rx=0 tx=0).stop 2024-06-18T02:30:45.102 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:43.691+0000 7f0224a02640 1 -- 172.21.15.1:0/2931518352 shutdown_connections 2024-06-18T02:30:45.102 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:43.691+0000 7f0224a02640 1 --2- 172.21.15.1:0/2931518352 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0220100010 0x7f0220100440 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:45.102 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:43.691+0000 7f0224a02640 1 -- 172.21.15.1:0/2931518352 >> 172.21.15.1:0/2931518352 conn(0x7f02200fbab0 msgr2=0x7f02200fdec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:30:45.102 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:43.691+0000 7f0224a02640 1 -- 172.21.15.1:0/2931518352 shutdown_connections 2024-06-18T02:30:45.102 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:43.691+0000 7f0224a02640 1 -- 172.21.15.1:0/2931518352 wait complete. 2024-06-18T02:30:45.102 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:43.692+0000 7f0224a02640 1 Processor -- start 2024-06-18T02:30:45.102 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:43.692+0000 7f0224a02640 1 -- start start 2024-06-18T02:30:45.103 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:43.692+0000 7f0224a02640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0220100010 0x7f0220112c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:45.103 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:43.692+0000 7f0224a02640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0220113180 con 0x7f0220100010 2024-06-18T02:30:45.103 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:43.692+0000 7f021ed9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0220100010 0x7f0220112c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:30:45.103 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:43.692+0000 7f021ed9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0220100010 0x7f0220112c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57810/0 (socket says 172.21.15.1:57810) 2024-06-18T02:30:45.103 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:43.692+0000 7f021ed9d640 1 -- 172.21.15.1:0/3948729052 learned_addr learned my addr 172.21.15.1:0/3948729052 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:30:45.103 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:43.692+0000 7f021ed9d640 1 -- 172.21.15.1:0/3948729052 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0210009450 con 0x7f0220100010 2024-06-18T02:30:45.103 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:43.693+0000 7f021ed9d640 1 --2- 172.21.15.1:0/3948729052 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0220100010 0x7f0220112c40 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f02100098a0 tx=0x7f021002c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:30:45.103 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:43.693+0000 7f02077fe640 1 -- 172.21.15.1:0/3948729052 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0210031e10 con 0x7f0220100010 2024-06-18T02:30:45.103 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:43.693+0000 7f02077fe640 1 -- 172.21.15.1:0/3948729052 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f021003b910 con 0x7f0220100010 2024-06-18T02:30:45.103 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:43.693+0000 7f0224a02640 1 -- 172.21.15.1:0/3948729052 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0220113380 con 0x7f0220100010 2024-06-18T02:30:45.103 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:43.693+0000 7f02077fe640 1 -- 172.21.15.1:0/3948729052 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f021003a880 con 0x7f0220100010 2024-06-18T02:30:45.103 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:43.693+0000 7f0224a02640 1 -- 172.21.15.1:0/3948729052 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02201137f0 con 0x7f0220100010 2024-06-18T02:30:45.103 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:43.694+0000 7f02077fe640 1 -- 172.21.15.1:0/3948729052 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f0210039070 con 0x7f0220100010 2024-06-18T02:30:45.104 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:43.694+0000 7f02077fe640 1 --2- 172.21.15.1:0/3948729052 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f0208034810 0x7f0208036cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:45.104 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:43.694+0000 7f02077fe640 1 -- 172.21.15.1:0/3948729052 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f0210066d90 con 0x7f0220100010 2024-06-18T02:30:45.104 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:43.694+0000 7f021e59c640 1 --2- 172.21.15.1:0/3948729052 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f0208034810 0x7f0208036cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:30:45.104 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:43.694+0000 7f0224a02640 1 -- 172.21.15.1:0/3948729052 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f022005ead0 con 0x7f0220100010 2024-06-18T02:30:45.104 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:43.694+0000 7f021e59c640 1 --2- 172.21.15.1:0/3948729052 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f0208034810 0x7f0208036cc0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f0214006fd0 tx=0x7f0214006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:30:45.104 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:43.699+0000 7f02077fe640 1 -- 172.21.15.1:0/3948729052 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f021003f020 con 0x7f0220100010 2024-06-18T02:30:45.104 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:43.890+0000 7f0224a02640 1 -- 172.21.15.1:0/3948729052 --> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f02200fceb0 con 0x7f0208034810 2024-06-18T02:30:45.104 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:43.894+0000 7f02077fe640 1 -- 172.21.15.1:0/3948729052 <== mgr.14118 v2:172.21.15.1:6800/4268462134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f02200fceb0 con 0x7f0208034810 2024-06-18T02:30:45.104 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:43.896+0000 7f0224a02640 1 -- 172.21.15.1:0/3948729052 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f0208034810 msgr2=0x7f0208036cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:30:45.104 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:43.896+0000 7f0224a02640 1 --2- 172.21.15.1:0/3948729052 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f0208034810 0x7f0208036cc0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f0214006fd0 tx=0x7f0214006db0 comp rx=0 tx=0).stop 2024-06-18T02:30:45.104 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:43.896+0000 7f0224a02640 1 -- 172.21.15.1:0/3948729052 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0220100010 msgr2=0x7f0220112c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:30:45.104 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:43.896+0000 7f0224a02640 1 --2- 172.21.15.1:0/3948729052 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0220100010 0x7f0220112c40 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f02100098a0 tx=0x7f021002c040 comp rx=0 tx=0).stop 2024-06-18T02:30:45.104 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:43.896+0000 7f0224a02640 1 -- 172.21.15.1:0/3948729052 shutdown_connections 2024-06-18T02:30:45.104 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:43.896+0000 7f0224a02640 1 --2- 172.21.15.1:0/3948729052 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f0208034810 0x7f0208036cc0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:45.105 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:43.896+0000 7f0224a02640 1 --2- 172.21.15.1:0/3948729052 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0220100010 0x7f0220112c40 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:45.105 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:43.896+0000 7f0224a02640 1 -- 172.21.15.1:0/3948729052 >> 172.21.15.1:0/3948729052 conn(0x7f02200fbab0 msgr2=0x7f02200fc6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:30:45.105 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:43.896+0000 7f0224a02640 1 -- 172.21.15.1:0/3948729052 shutdown_connections 2024-06-18T02:30:45.105 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:43.896+0000 7f0224a02640 1 -- 172.21.15.1:0/3948729052 wait complete. 2024-06-18T02:30:45.105 INFO:teuthology.orchestra.run.smithi001.stdout:Deploying prometheus service with default placement... 2024-06-18T02:30:45.377 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:45 smithi001 ceph-mon[29099]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-06-18T02:30:45.377 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:45 smithi001 ceph-mon[29099]: Saving service crash spec with placement * 2024-06-18T02:30:45.377 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:45 smithi001 ceph-mon[29099]: from='mgr.14118 172.21.15.1:0/2084661558' entity='mgr.smithi001.teoawz' 2024-06-18T02:30:46.333 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout Scheduled prometheus update... 2024-06-18T02:30:46.333 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:45.554+0000 7f002c062640 1 Processor -- start 2024-06-18T02:30:46.333 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:45.554+0000 7f002c062640 1 -- start start 2024-06-18T02:30:46.333 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:45.554+0000 7f002c062640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f002410c740 0x7f002410cb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:46.333 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:45.554+0000 7f002c062640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f002410d090 con 0x7f002410c740 2024-06-18T02:30:46.333 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:45.554+0000 7f0029dfe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f002410c740 0x7f002410cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:30:46.333 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:45.554+0000 7f0029dfe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f002410c740 0x7f002410cb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56776/0 (socket says 172.21.15.1:56776) 2024-06-18T02:30:46.333 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:45.554+0000 7f0029dfe640 1 -- 172.21.15.1:0/2963898267 learned_addr learned my addr 172.21.15.1:0/2963898267 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:30:46.333 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:45.555+0000 7f0029dfe640 1 -- 172.21.15.1:0/2963898267 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f002410d1d0 con 0x7f002410c740 2024-06-18T02:30:46.333 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:45.555+0000 7f0029dfe640 1 --2- 172.21.15.1:0/2963898267 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f002410c740 0x7f002410cb50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0020009840 tx=0x7f00200091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9aa137a732173713 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:30:46.333 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:45.556+0000 7f0028dfc640 1 -- 172.21.15.1:0/2963898267 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0020032440 con 0x7f002410c740 2024-06-18T02:30:46.333 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:45.556+0000 7f0028dfc640 1 -- 172.21.15.1:0/2963898267 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0020032a70 con 0x7f002410c740 2024-06-18T02:30:46.334 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:45.556+0000 7f0028dfc640 1 -- 172.21.15.1:0/2963898267 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0020031ad0 con 0x7f002410c740 2024-06-18T02:30:46.334 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:45.556+0000 7f002c062640 1 -- 172.21.15.1:0/2963898267 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f002410c740 msgr2=0x7f002410cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:30:46.334 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:45.556+0000 7f002c062640 1 --2- 172.21.15.1:0/2963898267 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f002410c740 0x7f002410cb50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0020009840 tx=0x7f00200091d0 comp rx=0 tx=0).stop 2024-06-18T02:30:46.334 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:45.556+0000 7f002c062640 1 -- 172.21.15.1:0/2963898267 shutdown_connections 2024-06-18T02:30:46.334 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:45.556+0000 7f002c062640 1 --2- 172.21.15.1:0/2963898267 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f002410c740 0x7f002410cb50 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:46.334 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:45.556+0000 7f002c062640 1 -- 172.21.15.1:0/2963898267 >> 172.21.15.1:0/2963898267 conn(0x7f00240683d0 msgr2=0x7f00240687f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:30:46.334 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:45.557+0000 7f002c062640 1 -- 172.21.15.1:0/2963898267 shutdown_connections 2024-06-18T02:30:46.334 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:45.557+0000 7f002c062640 1 -- 172.21.15.1:0/2963898267 wait complete. 2024-06-18T02:30:46.334 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:45.557+0000 7f002c062640 1 Processor -- start 2024-06-18T02:30:46.334 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:45.557+0000 7f002c062640 1 -- start start 2024-06-18T02:30:46.334 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:45.557+0000 7f002c062640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f002410c740 0x7f002411b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:46.334 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:45.557+0000 7f002c062640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f002411baf0 con 0x7f002410c740 2024-06-18T02:30:46.334 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:45.557+0000 7f0029dfe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f002410c740 0x7f002411b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:30:46.335 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:45.557+0000 7f0029dfe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f002410c740 0x7f002411b5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56784/0 (socket says 172.21.15.1:56784) 2024-06-18T02:30:46.335 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:45.557+0000 7f0029dfe640 1 -- 172.21.15.1:0/883391261 learned_addr learned my addr 172.21.15.1:0/883391261 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:30:46.335 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:45.558+0000 7f0029dfe640 1 -- 172.21.15.1:0/883391261 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0020009450 con 0x7f002410c740 2024-06-18T02:30:46.335 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:45.558+0000 7f0029dfe640 1 --2- 172.21.15.1:0/883391261 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f002410c740 0x7f002411b5b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f00200098a0 tx=0x7f002002c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:30:46.335 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:45.558+0000 7f001a7fc640 1 -- 172.21.15.1:0/883391261 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0020031e90 con 0x7f002410c740 2024-06-18T02:30:46.335 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:45.558+0000 7f001a7fc640 1 -- 172.21.15.1:0/883391261 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f002003b910 con 0x7f002410c740 2024-06-18T02:30:46.335 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:45.558+0000 7f001a7fc640 1 -- 172.21.15.1:0/883391261 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f002003a8d0 con 0x7f002410c740 2024-06-18T02:30:46.335 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:45.558+0000 7f002c062640 1 -- 172.21.15.1:0/883391261 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f002411bcf0 con 0x7f002410c740 2024-06-18T02:30:46.335 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:45.558+0000 7f002c062640 1 -- 172.21.15.1:0/883391261 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f002411c160 con 0x7f002410c740 2024-06-18T02:30:46.335 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:45.559+0000 7f001a7fc640 1 -- 172.21.15.1:0/883391261 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f0020039070 con 0x7f002410c740 2024-06-18T02:30:46.335 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:45.559+0000 7f001a7fc640 1 --2- 172.21.15.1:0/883391261 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f0010034860 0x7f0010036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:46.335 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:45.559+0000 7f001a7fc640 1 -- 172.21.15.1:0/883391261 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f0020066e90 con 0x7f002410c740 2024-06-18T02:30:46.335 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:45.559+0000 7f002c062640 1 -- 172.21.15.1:0/883391261 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f002405ead0 con 0x7f002410c740 2024-06-18T02:30:46.336 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:45.559+0000 7f00295fd640 1 --2- 172.21.15.1:0/883391261 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f0010034860 0x7f0010036d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:30:46.336 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:45.560+0000 7f00295fd640 1 --2- 172.21.15.1:0/883391261 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f0010034860 0x7f0010036d10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0014006fd0 tx=0x7f0014006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:30:46.336 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:45.564+0000 7f001a7fc640 1 -- 172.21.15.1:0/883391261 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f0020037070 con 0x7f002410c740 2024-06-18T02:30:46.336 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:45.750+0000 7f002c062640 1 -- 172.21.15.1:0/883391261 --> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f002410f330 con 0x7f0010034860 2024-06-18T02:30:46.336 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:45.754+0000 7f001a7fc640 1 -- 172.21.15.1:0/883391261 <== mgr.14118 v2:172.21.15.1:6800/4268462134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f002410f330 con 0x7f0010034860 2024-06-18T02:30:46.336 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:45.756+0000 7f002c062640 1 -- 172.21.15.1:0/883391261 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f0010034860 msgr2=0x7f0010036d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:30:46.336 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:45.756+0000 7f002c062640 1 --2- 172.21.15.1:0/883391261 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f0010034860 0x7f0010036d10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0014006fd0 tx=0x7f0014006db0 comp rx=0 tx=0).stop 2024-06-18T02:30:46.336 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:45.756+0000 7f002c062640 1 -- 172.21.15.1:0/883391261 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f002410c740 msgr2=0x7f002411b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:30:46.336 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:45.756+0000 7f002c062640 1 --2- 172.21.15.1:0/883391261 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f002410c740 0x7f002411b5b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f00200098a0 tx=0x7f002002c040 comp rx=0 tx=0).stop 2024-06-18T02:30:46.336 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:45.756+0000 7f002c062640 1 -- 172.21.15.1:0/883391261 shutdown_connections 2024-06-18T02:30:46.336 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:45.756+0000 7f002c062640 1 --2- 172.21.15.1:0/883391261 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f0010034860 0x7f0010036d10 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:46.336 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:45.756+0000 7f002c062640 1 --2- 172.21.15.1:0/883391261 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f002410c740 0x7f002411b5b0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:46.336 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:45.756+0000 7f002c062640 1 -- 172.21.15.1:0/883391261 >> 172.21.15.1:0/883391261 conn(0x7f00240683d0 msgr2=0x7f002410ec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:30:46.337 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:45.757+0000 7f002c062640 1 -- 172.21.15.1:0/883391261 shutdown_connections 2024-06-18T02:30:46.337 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:45.757+0000 7f002c062640 1 -- 172.21.15.1:0/883391261 wait complete. 2024-06-18T02:30:46.337 INFO:teuthology.orchestra.run.smithi001.stdout:Deploying grafana service with default placement... 2024-06-18T02:30:46.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:46 smithi001 ceph-mon[29099]: from='mgr.14118 172.21.15.1:0/2084661558' entity='mgr.smithi001.teoawz' 2024-06-18T02:30:46.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:46 smithi001 ceph-mon[29099]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-06-18T02:30:46.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:46 smithi001 ceph-mon[29099]: Saving service prometheus spec with placement count:1 2024-06-18T02:30:46.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:46 smithi001 ceph-mon[29099]: from='mgr.14118 172.21.15.1:0/2084661558' entity='mgr.smithi001.teoawz' 2024-06-18T02:30:47.485 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout Scheduled grafana update... 2024-06-18T02:30:47.485 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:46.704+0000 7f841e1fa640 1 Processor -- start 2024-06-18T02:30:47.485 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:46.704+0000 7f841e1fa640 1 -- start start 2024-06-18T02:30:47.486 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:46.704+0000 7f841e1fa640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84181031a0 0x7f84181035b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:47.486 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:46.704+0000 7f841e1fa640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8418103b80 con 0x7f84181031a0 2024-06-18T02:30:47.486 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:46.704+0000 7f84177fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84181031a0 0x7f84181035b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:30:47.486 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:46.704+0000 7f84177fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84181031a0 0x7f84181035b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56786/0 (socket says 172.21.15.1:56786) 2024-06-18T02:30:47.486 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:46.704+0000 7f84177fe640 1 -- 172.21.15.1:0/1873037107 learned_addr learned my addr 172.21.15.1:0/1873037107 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:30:47.486 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:46.705+0000 7f84177fe640 1 -- 172.21.15.1:0/1873037107 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8418103cc0 con 0x7f84181031a0 2024-06-18T02:30:47.486 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:46.705+0000 7f84177fe640 1 --2- 172.21.15.1:0/1873037107 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84181031a0 0x7f84181035b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8408009840 tx=0x7f84080091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa33af15c58cb124 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:30:47.486 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:46.706+0000 7f84167fc640 1 -- 172.21.15.1:0/1873037107 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8408032440 con 0x7f84181031a0 2024-06-18T02:30:47.486 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:46.706+0000 7f84167fc640 1 -- 172.21.15.1:0/1873037107 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8408032a70 con 0x7f84181031a0 2024-06-18T02:30:47.487 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:46.706+0000 7f84167fc640 1 -- 172.21.15.1:0/1873037107 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8408031ad0 con 0x7f84181031a0 2024-06-18T02:30:47.487 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:46.706+0000 7f841e1fa640 1 -- 172.21.15.1:0/1873037107 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84181031a0 msgr2=0x7f84181035b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:30:47.487 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:46.706+0000 7f841e1fa640 1 --2- 172.21.15.1:0/1873037107 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84181031a0 0x7f84181035b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8408009840 tx=0x7f84080091d0 comp rx=0 tx=0).stop 2024-06-18T02:30:47.487 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:46.706+0000 7f841e1fa640 1 -- 172.21.15.1:0/1873037107 shutdown_connections 2024-06-18T02:30:47.487 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:46.706+0000 7f841e1fa640 1 --2- 172.21.15.1:0/1873037107 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84181031a0 0x7f84181035b0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:47.487 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:46.706+0000 7f841e1fa640 1 -- 172.21.15.1:0/1873037107 >> 172.21.15.1:0/1873037107 conn(0x7f84180fbc10 msgr2=0x7f84180fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:30:47.487 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:46.706+0000 7f841e1fa640 1 -- 172.21.15.1:0/1873037107 shutdown_connections 2024-06-18T02:30:47.487 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:46.706+0000 7f841e1fa640 1 -- 172.21.15.1:0/1873037107 wait complete. 2024-06-18T02:30:47.487 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:46.707+0000 7f841e1fa640 1 Processor -- start 2024-06-18T02:30:47.488 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:46.707+0000 7f841e1fa640 1 -- start start 2024-06-18T02:30:47.488 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:46.707+0000 7f841e1fa640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84181031a0 0x7f8418110c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:47.488 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:46.707+0000 7f841e1fa640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8418111160 con 0x7f84181031a0 2024-06-18T02:30:47.488 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:46.707+0000 7f84177fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84181031a0 0x7f8418110c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:30:47.488 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:46.707+0000 7f84177fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84181031a0 0x7f8418110c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56790/0 (socket says 172.21.15.1:56790) 2024-06-18T02:30:47.488 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:46.707+0000 7f84177fe640 1 -- 172.21.15.1:0/395901485 learned_addr learned my addr 172.21.15.1:0/395901485 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:30:47.488 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:46.708+0000 7f84177fe640 1 -- 172.21.15.1:0/395901485 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8408009450 con 0x7f84181031a0 2024-06-18T02:30:47.488 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:46.708+0000 7f84177fe640 1 --2- 172.21.15.1:0/395901485 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84181031a0 0x7f8418110c20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f84080098a0 tx=0x7f840802c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:30:47.488 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:46.708+0000 7f83fffff640 1 -- 172.21.15.1:0/395901485 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8408031e90 con 0x7f84181031a0 2024-06-18T02:30:47.489 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:46.708+0000 7f83fffff640 1 -- 172.21.15.1:0/395901485 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f840803b910 con 0x7f84181031a0 2024-06-18T02:30:47.489 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:46.708+0000 7f83fffff640 1 -- 172.21.15.1:0/395901485 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f840803a8d0 con 0x7f84181031a0 2024-06-18T02:30:47.489 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:46.708+0000 7f841e1fa640 1 -- 172.21.15.1:0/395901485 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8418111360 con 0x7f84181031a0 2024-06-18T02:30:47.489 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:46.708+0000 7f841e1fa640 1 -- 172.21.15.1:0/395901485 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84181117d0 con 0x7f84181031a0 2024-06-18T02:30:47.489 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:46.709+0000 7f83fffff640 1 -- 172.21.15.1:0/395901485 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f8408039070 con 0x7f84181031a0 2024-06-18T02:30:47.489 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:46.709+0000 7f83fffff640 1 --2- 172.21.15.1:0/395901485 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f8400034810 0x7f8400036cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:47.489 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:46.709+0000 7f83fffff640 1 -- 172.21.15.1:0/395901485 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f8408066d80 con 0x7f84181031a0 2024-06-18T02:30:47.489 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:46.709+0000 7f8416ffd640 1 --2- 172.21.15.1:0/395901485 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f8400034810 0x7f8400036cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:30:47.489 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:46.709+0000 7f841e1fa640 1 -- 172.21.15.1:0/395901485 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f841805ead0 con 0x7f84181031a0 2024-06-18T02:30:47.489 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:46.709+0000 7f8416ffd640 1 --2- 172.21.15.1:0/395901485 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f8400034810 0x7f8400036cc0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f840c006fd0 tx=0x7f840c006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:30:47.490 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:46.714+0000 7f83fffff640 1 -- 172.21.15.1:0/395901485 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f840803f020 con 0x7f84181031a0 2024-06-18T02:30:47.490 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:46.903+0000 7f841e1fa640 1 -- 172.21.15.1:0/395901485 --> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f84180fcf50 con 0x7f8400034810 2024-06-18T02:30:47.490 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:46.907+0000 7f83fffff640 1 -- 172.21.15.1:0/395901485 <== mgr.14118 v2:172.21.15.1:6800/4268462134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f84180fcf50 con 0x7f8400034810 2024-06-18T02:30:47.490 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:46.910+0000 7f841e1fa640 1 -- 172.21.15.1:0/395901485 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f8400034810 msgr2=0x7f8400036cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:30:47.490 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:46.910+0000 7f841e1fa640 1 --2- 172.21.15.1:0/395901485 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f8400034810 0x7f8400036cc0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f840c006fd0 tx=0x7f840c006db0 comp rx=0 tx=0).stop 2024-06-18T02:30:47.490 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:46.910+0000 7f841e1fa640 1 -- 172.21.15.1:0/395901485 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84181031a0 msgr2=0x7f8418110c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:30:47.490 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:46.910+0000 7f841e1fa640 1 --2- 172.21.15.1:0/395901485 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84181031a0 0x7f8418110c20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f84080098a0 tx=0x7f840802c040 comp rx=0 tx=0).stop 2024-06-18T02:30:47.491 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:46.910+0000 7f841e1fa640 1 -- 172.21.15.1:0/395901485 shutdown_connections 2024-06-18T02:30:47.491 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:46.910+0000 7f841e1fa640 1 --2- 172.21.15.1:0/395901485 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f8400034810 0x7f8400036cc0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:47.491 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:46.910+0000 7f841e1fa640 1 --2- 172.21.15.1:0/395901485 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f84181031a0 0x7f8418110c20 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:47.491 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:46.910+0000 7f841e1fa640 1 -- 172.21.15.1:0/395901485 >> 172.21.15.1:0/395901485 conn(0x7f84180fbc10 msgr2=0x7f84180fc820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:30:47.493 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:46.910+0000 7f841e1fa640 1 -- 172.21.15.1:0/395901485 shutdown_connections 2024-06-18T02:30:47.494 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:46.910+0000 7f841e1fa640 1 -- 172.21.15.1:0/395901485 wait complete. 2024-06-18T02:30:47.494 INFO:teuthology.orchestra.run.smithi001.stdout:Deploying node-exporter service with default placement... 2024-06-18T02:30:48.058 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:47 smithi001 ceph-mon[29099]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-06-18T02:30:48.059 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:47 smithi001 ceph-mon[29099]: Saving service grafana spec with placement count:1 2024-06-18T02:30:48.059 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:47 smithi001 ceph-mon[29099]: from='mgr.14118 172.21.15.1:0/2084661558' entity='mgr.smithi001.teoawz' 2024-06-18T02:30:49.169 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:49 smithi001 ceph-mon[29099]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-06-18T02:30:49.169 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:49 smithi001 ceph-mon[29099]: Saving service node-exporter spec with placement * 2024-06-18T02:30:49.169 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:49 smithi001 ceph-mon[29099]: from='mgr.14118 172.21.15.1:0/2084661558' entity='mgr.smithi001.teoawz' 2024-06-18T02:30:49.178 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout Scheduled node-exporter update... 2024-06-18T02:30:49.178 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:47.866+0000 7f165e633640 1 Processor -- start 2024-06-18T02:30:49.178 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:47.866+0000 7f165e633640 1 -- start start 2024-06-18T02:30:49.178 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:47.867+0000 7f165e633640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1658075b00 0x7f1658106df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:49.178 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:47.867+0000 7f165e633640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1658075f10 con 0x7f1658075b00 2024-06-18T02:30:49.178 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:47.867+0000 7f1657fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1658075b00 0x7f1658106df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:30:49.178 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:47.867+0000 7f1657fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1658075b00 0x7f1658106df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56798/0 (socket says 172.21.15.1:56798) 2024-06-18T02:30:49.178 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:47.867+0000 7f1657fff640 1 -- 172.21.15.1:0/3233873438 learned_addr learned my addr 172.21.15.1:0/3233873438 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:30:49.178 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:47.867+0000 7f1657fff640 1 -- 172.21.15.1:0/3233873438 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16581027e0 con 0x7f1658075b00 2024-06-18T02:30:49.179 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:47.868+0000 7f1657fff640 1 --2- 172.21.15.1:0/3233873438 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1658075b00 0x7f1658106df0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1648009840 tx=0x7f16480091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90469426d23927cf server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:30:49.179 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:47.868+0000 7f16577fe640 1 -- 172.21.15.1:0/3233873438 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1648032440 con 0x7f1658075b00 2024-06-18T02:30:49.179 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:47.868+0000 7f16577fe640 1 -- 172.21.15.1:0/3233873438 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1648032a70 con 0x7f1658075b00 2024-06-18T02:30:49.179 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:47.868+0000 7f16577fe640 1 -- 172.21.15.1:0/3233873438 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1648031ad0 con 0x7f1658075b00 2024-06-18T02:30:49.179 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:47.869+0000 7f165e633640 1 -- 172.21.15.1:0/3233873438 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1658075b00 msgr2=0x7f1658106df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:30:49.179 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:47.869+0000 7f165e633640 1 --2- 172.21.15.1:0/3233873438 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1658075b00 0x7f1658106df0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1648009840 tx=0x7f16480091d0 comp rx=0 tx=0).stop 2024-06-18T02:30:49.180 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:47.869+0000 7f165e633640 1 -- 172.21.15.1:0/3233873438 shutdown_connections 2024-06-18T02:30:49.180 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:47.869+0000 7f165e633640 1 --2- 172.21.15.1:0/3233873438 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1658075b00 0x7f1658106df0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:49.180 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:47.869+0000 7f165e633640 1 -- 172.21.15.1:0/3233873438 >> 172.21.15.1:0/3233873438 conn(0x7f16580fc460 msgr2=0x7f16580fe870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:30:49.180 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:47.869+0000 7f165e633640 1 -- 172.21.15.1:0/3233873438 shutdown_connections 2024-06-18T02:30:49.180 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:47.869+0000 7f165e633640 1 -- 172.21.15.1:0/3233873438 wait complete. 2024-06-18T02:30:49.180 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:47.869+0000 7f165e633640 1 Processor -- start 2024-06-18T02:30:49.180 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:47.869+0000 7f165e633640 1 -- start start 2024-06-18T02:30:49.180 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:47.869+0000 7f165e633640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1658075b00 0x7f1658112df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:49.180 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:47.870+0000 7f165e633640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1658113330 con 0x7f1658075b00 2024-06-18T02:30:49.180 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:47.870+0000 7f1657fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1658075b00 0x7f1658112df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:30:49.181 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:47.870+0000 7f1657fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1658075b00 0x7f1658112df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56802/0 (socket says 172.21.15.1:56802) 2024-06-18T02:30:49.181 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:47.870+0000 7f1657fff640 1 -- 172.21.15.1:0/1125964079 learned_addr learned my addr 172.21.15.1:0/1125964079 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:30:49.181 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:47.870+0000 7f1657fff640 1 -- 172.21.15.1:0/1125964079 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1648009450 con 0x7f1658075b00 2024-06-18T02:30:49.181 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:47.870+0000 7f1657fff640 1 --2- 172.21.15.1:0/1125964079 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1658075b00 0x7f1658112df0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f16480098a0 tx=0x7f164802c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:30:49.181 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:47.870+0000 7f16557fa640 1 -- 172.21.15.1:0/1125964079 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1648031e90 con 0x7f1658075b00 2024-06-18T02:30:49.181 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:47.870+0000 7f16557fa640 1 -- 172.21.15.1:0/1125964079 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f164803b910 con 0x7f1658075b00 2024-06-18T02:30:49.181 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:47.870+0000 7f16557fa640 1 -- 172.21.15.1:0/1125964079 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f164803a9b0 con 0x7f1658075b00 2024-06-18T02:30:49.181 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:47.870+0000 7f165e633640 1 -- 172.21.15.1:0/1125964079 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1658113590 con 0x7f1658075b00 2024-06-18T02:30:49.181 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:47.870+0000 7f165e633640 1 -- 172.21.15.1:0/1125964079 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1658113f80 con 0x7f1658075b00 2024-06-18T02:30:49.181 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:47.871+0000 7f16557fa640 1 -- 172.21.15.1:0/1125964079 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f1648039070 con 0x7f1658075b00 2024-06-18T02:30:49.181 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:47.871+0000 7f165e633640 1 -- 172.21.15.1:0/1125964079 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f165804b480 con 0x7f1658075b00 2024-06-18T02:30:49.181 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:47.871+0000 7f16557fa640 1 --2- 172.21.15.1:0/1125964079 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f1644034860 0x7f1644036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:49.181 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:47.871+0000 7f16557fa640 1 -- 172.21.15.1:0/1125964079 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f1648066da0 con 0x7f1658075b00 2024-06-18T02:30:49.182 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:47.872+0000 7f164ffff640 1 --2- 172.21.15.1:0/1125964079 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f1644034860 0x7f1644036d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:30:49.182 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:47.872+0000 7f164ffff640 1 --2- 172.21.15.1:0/1125964079 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f1644034860 0x7f1644036d10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f1640006fd0 tx=0x7f1640006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:30:49.182 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:47.876+0000 7f16557fa640 1 -- 172.21.15.1:0/1125964079 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f1648039500 con 0x7f1658075b00 2024-06-18T02:30:49.182 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:48.059+0000 7f165e633640 1 -- 172.21.15.1:0/1125964079 --> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f16580fd900 con 0x7f1644034860 2024-06-18T02:30:49.182 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:48.063+0000 7f16557fa640 1 -- 172.21.15.1:0/1125964079 <== mgr.14118 v2:172.21.15.1:6800/4268462134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f16580fd900 con 0x7f1644034860 2024-06-18T02:30:49.182 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:48.065+0000 7f165e633640 1 -- 172.21.15.1:0/1125964079 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f1644034860 msgr2=0x7f1644036d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:30:49.182 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:48.065+0000 7f165e633640 1 --2- 172.21.15.1:0/1125964079 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f1644034860 0x7f1644036d10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f1640006fd0 tx=0x7f1640006db0 comp rx=0 tx=0).stop 2024-06-18T02:30:49.182 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:48.065+0000 7f165e633640 1 -- 172.21.15.1:0/1125964079 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1658075b00 msgr2=0x7f1658112df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:30:49.182 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:48.065+0000 7f165e633640 1 --2- 172.21.15.1:0/1125964079 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1658075b00 0x7f1658112df0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f16480098a0 tx=0x7f164802c040 comp rx=0 tx=0).stop 2024-06-18T02:30:49.182 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:48.065+0000 7f165e633640 1 -- 172.21.15.1:0/1125964079 shutdown_connections 2024-06-18T02:30:49.182 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:48.065+0000 7f165e633640 1 --2- 172.21.15.1:0/1125964079 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f1644034860 0x7f1644036d10 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:49.183 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:48.065+0000 7f165e633640 1 --2- 172.21.15.1:0/1125964079 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1658075b00 0x7f1658112df0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:49.183 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:48.065+0000 7f165e633640 1 -- 172.21.15.1:0/1125964079 >> 172.21.15.1:0/1125964079 conn(0x7f16580fc460 msgr2=0x7f16580fd110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:30:49.183 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:48.065+0000 7f165e633640 1 -- 172.21.15.1:0/1125964079 shutdown_connections 2024-06-18T02:30:49.183 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:48.065+0000 7f165e633640 1 -- 172.21.15.1:0/1125964079 wait complete. 2024-06-18T02:30:49.183 INFO:teuthology.orchestra.run.smithi001.stdout:Deploying alertmanager service with default placement... 2024-06-18T02:30:50.369 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout Scheduled alertmanager update... 2024-06-18T02:30:50.375 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:49.643+0000 7f7e1f1b7640 1 Processor -- start 2024-06-18T02:30:50.375 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:49.643+0000 7f7e1f1b7640 1 -- start start 2024-06-18T02:30:50.375 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:49.643+0000 7f7e1f1b7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e18075ac0 0x7f7e18104a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:50.375 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:49.643+0000 7f7e1f1b7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e18075ed0 con 0x7f7e18075ac0 2024-06-18T02:30:50.375 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:49.643+0000 7f7e1cf53640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e18075ac0 0x7f7e18104a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:30:50.375 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:49.643+0000 7f7e1cf53640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e18075ac0 0x7f7e18104a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56804/0 (socket says 172.21.15.1:56804) 2024-06-18T02:30:50.375 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:49.643+0000 7f7e1cf53640 1 -- 172.21.15.1:0/3031929167 learned_addr learned my addr 172.21.15.1:0/3031929167 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:30:50.375 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:49.644+0000 7f7e1cf53640 1 -- 172.21.15.1:0/3031929167 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e18104f70 con 0x7f7e18075ac0 2024-06-18T02:30:50.375 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:49.644+0000 7f7e1cf53640 1 --2- 172.21.15.1:0/3031929167 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e18075ac0 0x7f7e18104a30 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c009840 tx=0x7f7e0c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2c9502032344dd4 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:30:50.375 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:49.645+0000 7f7e177fe640 1 -- 172.21.15.1:0/3031929167 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e0c032440 con 0x7f7e18075ac0 2024-06-18T02:30:50.375 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:49.645+0000 7f7e177fe640 1 -- 172.21.15.1:0/3031929167 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7e0c032a70 con 0x7f7e18075ac0 2024-06-18T02:30:50.376 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:49.645+0000 7f7e177fe640 1 -- 172.21.15.1:0/3031929167 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e0c031ad0 con 0x7f7e18075ac0 2024-06-18T02:30:50.376 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:49.645+0000 7f7e1f1b7640 1 -- 172.21.15.1:0/3031929167 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e18075ac0 msgr2=0x7f7e18104a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:30:50.376 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:49.645+0000 7f7e1f1b7640 1 --2- 172.21.15.1:0/3031929167 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e18075ac0 0x7f7e18104a30 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c009840 tx=0x7f7e0c0091d0 comp rx=0 tx=0).stop 2024-06-18T02:30:50.376 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:49.646+0000 7f7e1f1b7640 1 -- 172.21.15.1:0/3031929167 shutdown_connections 2024-06-18T02:30:50.376 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:49.646+0000 7f7e1f1b7640 1 --2- 172.21.15.1:0/3031929167 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e18075ac0 0x7f7e18104a30 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:50.376 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:49.646+0000 7f7e1f1b7640 1 -- 172.21.15.1:0/3031929167 >> 172.21.15.1:0/3031929167 conn(0x7f7e180fc460 msgr2=0x7f7e180fe870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:30:50.376 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:49.646+0000 7f7e1f1b7640 1 -- 172.21.15.1:0/3031929167 shutdown_connections 2024-06-18T02:30:50.376 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:49.646+0000 7f7e1f1b7640 1 -- 172.21.15.1:0/3031929167 wait complete. 2024-06-18T02:30:50.376 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:49.646+0000 7f7e1f1b7640 1 Processor -- start 2024-06-18T02:30:50.376 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:49.646+0000 7f7e1f1b7640 1 -- start start 2024-06-18T02:30:50.376 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:49.647+0000 7f7e1f1b7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e18075ac0 0x7f7e18110b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:50.376 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:49.647+0000 7f7e1f1b7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e181110d0 con 0x7f7e18075ac0 2024-06-18T02:30:50.376 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:49.647+0000 7f7e1cf53640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e18075ac0 0x7f7e18110b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:30:50.377 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:49.647+0000 7f7e1cf53640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e18075ac0 0x7f7e18110b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56814/0 (socket says 172.21.15.1:56814) 2024-06-18T02:30:50.377 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:49.647+0000 7f7e1cf53640 1 -- 172.21.15.1:0/9227929 learned_addr learned my addr 172.21.15.1:0/9227929 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:30:50.377 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:49.647+0000 7f7e1cf53640 1 -- 172.21.15.1:0/9227929 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e0c009450 con 0x7f7e18075ac0 2024-06-18T02:30:50.377 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:49.647+0000 7f7e1cf53640 1 --2- 172.21.15.1:0/9227929 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e18075ac0 0x7f7e18110b90 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c0317f0 tx=0x7f7e0c02c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:30:50.377 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:49.647+0000 7f7e157fa640 1 -- 172.21.15.1:0/9227929 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e0c032900 con 0x7f7e18075ac0 2024-06-18T02:30:50.377 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:49.647+0000 7f7e157fa640 1 -- 172.21.15.1:0/9227929 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7e0c03b910 con 0x7f7e18075ac0 2024-06-18T02:30:50.377 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:49.647+0000 7f7e1f1b7640 1 -- 172.21.15.1:0/9227929 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e181112d0 con 0x7f7e18075ac0 2024-06-18T02:30:50.377 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:49.648+0000 7f7e1f1b7640 1 -- 172.21.15.1:0/9227929 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e18111740 con 0x7f7e18075ac0 2024-06-18T02:30:50.377 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:49.648+0000 7f7e157fa640 1 -- 172.21.15.1:0/9227929 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e0c03a960 con 0x7f7e18075ac0 2024-06-18T02:30:50.377 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:49.648+0000 7f7e157fa640 1 -- 172.21.15.1:0/9227929 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f7e0c039070 con 0x7f7e18075ac0 2024-06-18T02:30:50.377 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:49.648+0000 7f7e157fa640 1 --2- 172.21.15.1:0/9227929 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f7e08034860 0x7f7e08036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:50.377 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:49.648+0000 7f7e157fa640 1 -- 172.21.15.1:0/9227929 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f7e0c067e50 con 0x7f7e18075ac0 2024-06-18T02:30:50.377 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:49.649+0000 7f7e17fff640 1 --2- 172.21.15.1:0/9227929 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f7e08034860 0x7f7e08036d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:30:50.378 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:49.649+0000 7f7e1f1b7640 1 -- 172.21.15.1:0/9227929 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e1805ead0 con 0x7f7e18075ac0 2024-06-18T02:30:50.378 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:49.649+0000 7f7e17fff640 1 --2- 172.21.15.1:0/9227929 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f7e08034860 0x7f7e08036d10 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f7e04006fd0 tx=0x7f7e04006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:30:50.378 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:49.653+0000 7f7e157fa640 1 -- 172.21.15.1:0/9227929 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f7e0c03f020 con 0x7f7e18075ac0 2024-06-18T02:30:50.378 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:49.837+0000 7f7e1f1b7640 1 -- 172.21.15.1:0/9227929 --> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7f7e180fd890 con 0x7f7e08034860 2024-06-18T02:30:50.378 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:49.842+0000 7f7e157fa640 1 -- 172.21.15.1:0/9227929 <== mgr.14118 v2:172.21.15.1:6800/4268462134 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7f7e180fd890 con 0x7f7e08034860 2024-06-18T02:30:50.378 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:49.843+0000 7f7e1f1b7640 1 -- 172.21.15.1:0/9227929 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f7e08034860 msgr2=0x7f7e08036d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:30:50.378 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:49.843+0000 7f7e1f1b7640 1 --2- 172.21.15.1:0/9227929 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f7e08034860 0x7f7e08036d10 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f7e04006fd0 tx=0x7f7e04006db0 comp rx=0 tx=0).stop 2024-06-18T02:30:50.378 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:49.843+0000 7f7e1f1b7640 1 -- 172.21.15.1:0/9227929 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e18075ac0 msgr2=0x7f7e18110b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:30:50.378 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:49.843+0000 7f7e1f1b7640 1 --2- 172.21.15.1:0/9227929 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e18075ac0 0x7f7e18110b90 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c0317f0 tx=0x7f7e0c02c040 comp rx=0 tx=0).stop 2024-06-18T02:30:50.378 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:49.844+0000 7f7e1f1b7640 1 -- 172.21.15.1:0/9227929 shutdown_connections 2024-06-18T02:30:50.378 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:49.844+0000 7f7e1f1b7640 1 --2- 172.21.15.1:0/9227929 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f7e08034860 0x7f7e08036d10 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:50.378 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:49.844+0000 7f7e1f1b7640 1 --2- 172.21.15.1:0/9227929 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7e18075ac0 0x7f7e18110b90 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:50.378 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:49.844+0000 7f7e1f1b7640 1 -- 172.21.15.1:0/9227929 >> 172.21.15.1:0/9227929 conn(0x7f7e180fc460 msgr2=0x7f7e180fd0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:30:50.379 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:49.844+0000 7f7e1f1b7640 1 -- 172.21.15.1:0/9227929 shutdown_connections 2024-06-18T02:30:50.379 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:49.844+0000 7f7e1f1b7640 1 -- 172.21.15.1:0/9227929 wait complete. 2024-06-18T02:30:50.495 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:50 smithi001 ceph-mon[29099]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:30:50.496 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:50 smithi001 ceph-mon[29099]: from='mgr.14118 172.21.15.1:0/2084661558' entity='mgr.smithi001.teoawz' 2024-06-18T02:30:51.507 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:51 smithi001 ceph-mon[29099]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-06-18T02:30:51.507 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:51 smithi001 ceph-mon[29099]: Saving service alertmanager spec with placement count:1 2024-06-18T02:30:51.507 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:51 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/627400669' entity='client.admin' 2024-06-18T02:30:51.515 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:50.768+0000 7f0ea25fb640 1 Processor -- start 2024-06-18T02:30:51.515 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:50.768+0000 7f0ea25fb640 1 -- start start 2024-06-18T02:30:51.515 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:50.768+0000 7f0ea25fb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0e9c101d00 0x7f0e9c102110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:51.515 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:50.768+0000 7f0ea25fb640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e9c102650 con 0x7f0e9c101d00 2024-06-18T02:30:51.515 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:50.768+0000 7f0e9bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0e9c101d00 0x7f0e9c102110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:30:51.515 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:50.768+0000 7f0e9bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0e9c101d00 0x7f0e9c102110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56830/0 (socket says 172.21.15.1:56830) 2024-06-18T02:30:51.515 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:50.768+0000 7f0e9bfff640 1 -- 172.21.15.1:0/3644942898 learned_addr learned my addr 172.21.15.1:0/3644942898 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:30:51.515 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:50.769+0000 7f0e9bfff640 1 -- 172.21.15.1:0/3644942898 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e9c102790 con 0x7f0e9c101d00 2024-06-18T02:30:51.515 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:50.769+0000 7f0e9bfff640 1 --2- 172.21.15.1:0/3644942898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0e9c101d00 0x7f0e9c102110 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c009840 tx=0x7f0e8c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cd0e4bd99706cab server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:30:51.515 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:50.770+0000 7f0e9affd640 1 -- 172.21.15.1:0/3644942898 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e8c032440 con 0x7f0e9c101d00 2024-06-18T02:30:51.516 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:50.770+0000 7f0e9affd640 1 -- 172.21.15.1:0/3644942898 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0e8c032a70 con 0x7f0e9c101d00 2024-06-18T02:30:51.516 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:50.770+0000 7f0e9affd640 1 -- 172.21.15.1:0/3644942898 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e8c031ad0 con 0x7f0e9c101d00 2024-06-18T02:30:51.516 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:50.770+0000 7f0ea25fb640 1 -- 172.21.15.1:0/3644942898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0e9c101d00 msgr2=0x7f0e9c102110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:30:51.516 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:50.770+0000 7f0ea25fb640 1 --2- 172.21.15.1:0/3644942898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0e9c101d00 0x7f0e9c102110 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c009840 tx=0x7f0e8c0091d0 comp rx=0 tx=0).stop 2024-06-18T02:30:51.516 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:50.770+0000 7f0ea25fb640 1 -- 172.21.15.1:0/3644942898 shutdown_connections 2024-06-18T02:30:51.516 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:50.770+0000 7f0ea25fb640 1 --2- 172.21.15.1:0/3644942898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0e9c101d00 0x7f0e9c102110 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:51.516 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:50.770+0000 7f0ea25fb640 1 -- 172.21.15.1:0/3644942898 >> 172.21.15.1:0/3644942898 conn(0x7f0e9c0fd500 msgr2=0x7f0e9c0ff910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:30:51.516 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:50.770+0000 7f0ea25fb640 1 -- 172.21.15.1:0/3644942898 shutdown_connections 2024-06-18T02:30:51.516 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:50.770+0000 7f0ea25fb640 1 -- 172.21.15.1:0/3644942898 wait complete. 2024-06-18T02:30:51.516 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:50.771+0000 7f0ea25fb640 1 Processor -- start 2024-06-18T02:30:51.516 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:50.771+0000 7f0ea25fb640 1 -- start start 2024-06-18T02:30:51.516 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:50.771+0000 7f0ea25fb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0e9c101d00 0x7f0e9c110ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:51.516 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:50.771+0000 7f0ea25fb640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e9c111420 con 0x7f0e9c101d00 2024-06-18T02:30:51.516 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:50.771+0000 7f0e9bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0e9c101d00 0x7f0e9c110ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:30:51.517 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:50.772+0000 7f0e9bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0e9c101d00 0x7f0e9c110ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56840/0 (socket says 172.21.15.1:56840) 2024-06-18T02:30:51.517 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:50.772+0000 7f0e9bfff640 1 -- 172.21.15.1:0/627400669 learned_addr learned my addr 172.21.15.1:0/627400669 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:30:51.517 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:50.772+0000 7f0e9bfff640 1 -- 172.21.15.1:0/627400669 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e8c009450 con 0x7f0e9c101d00 2024-06-18T02:30:51.517 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:50.772+0000 7f0e9bfff640 1 --2- 172.21.15.1:0/627400669 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0e9c101d00 0x7f0e9c110ee0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c009810 tx=0x7f0e8c02c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:30:51.517 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:50.772+0000 7f0e98ff9640 1 -- 172.21.15.1:0/627400669 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e8c0328b0 con 0x7f0e9c101d00 2024-06-18T02:30:51.517 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:50.772+0000 7f0e98ff9640 1 -- 172.21.15.1:0/627400669 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0e8c03b910 con 0x7f0e9c101d00 2024-06-18T02:30:51.517 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:50.772+0000 7f0ea25fb640 1 -- 172.21.15.1:0/627400669 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e9c111620 con 0x7f0e9c101d00 2024-06-18T02:30:51.517 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:50.772+0000 7f0e98ff9640 1 -- 172.21.15.1:0/627400669 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e8c03a980 con 0x7f0e9c101d00 2024-06-18T02:30:51.517 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:50.772+0000 7f0ea25fb640 1 -- 172.21.15.1:0/627400669 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e9c1120d0 con 0x7f0e9c101d00 2024-06-18T02:30:51.517 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:50.773+0000 7f0e98ff9640 1 -- 172.21.15.1:0/627400669 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f0e8c039070 con 0x7f0e9c101d00 2024-06-18T02:30:51.517 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:50.773+0000 7f0e98ff9640 1 --2- 172.21.15.1:0/627400669 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f0e84034810 0x7f0e84036cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:51.518 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:50.773+0000 7f0e98ff9640 1 -- 172.21.15.1:0/627400669 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f0e8c066f10 con 0x7f0e9c101d00 2024-06-18T02:30:51.518 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:50.773+0000 7f0e9b7fe640 1 --2- 172.21.15.1:0/627400669 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f0e84034810 0x7f0e84036cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:30:51.518 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:50.773+0000 7f0ea25fb640 1 -- 172.21.15.1:0/627400669 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e9c05ead0 con 0x7f0e9c101d00 2024-06-18T02:30:51.518 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:50.774+0000 7f0e9b7fe640 1 --2- 172.21.15.1:0/627400669 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f0e84034810 0x7f0e84036cc0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0e90006fd0 tx=0x7f0e90006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:30:51.518 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:50.778+0000 7f0e98ff9640 1 -- 172.21.15.1:0/627400669 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f0e8c03f080 con 0x7f0e9c101d00 2024-06-18T02:30:51.518 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:50.953+0000 7f0ea25fb640 1 -- 172.21.15.1:0/627400669 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f0e9c103c80 con 0x7f0e9c101d00 2024-06-18T02:30:51.518 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:50.956+0000 7f0e98ff9640 1 -- 172.21.15.1:0/627400669 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f0e8c0650d0 con 0x7f0e9c101d00 2024-06-18T02:30:51.518 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:50.958+0000 7f0ea25fb640 1 -- 172.21.15.1:0/627400669 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f0e84034810 msgr2=0x7f0e84036cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:30:51.518 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:50.958+0000 7f0ea25fb640 1 --2- 172.21.15.1:0/627400669 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f0e84034810 0x7f0e84036cc0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0e90006fd0 tx=0x7f0e90006db0 comp rx=0 tx=0).stop 2024-06-18T02:30:51.518 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:50.958+0000 7f0ea25fb640 1 -- 172.21.15.1:0/627400669 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0e9c101d00 msgr2=0x7f0e9c110ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:30:51.518 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:50.958+0000 7f0ea25fb640 1 --2- 172.21.15.1:0/627400669 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0e9c101d00 0x7f0e9c110ee0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c009810 tx=0x7f0e8c02c040 comp rx=0 tx=0).stop 2024-06-18T02:30:51.518 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:50.958+0000 7f0ea25fb640 1 -- 172.21.15.1:0/627400669 shutdown_connections 2024-06-18T02:30:51.519 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:50.958+0000 7f0ea25fb640 1 --2- 172.21.15.1:0/627400669 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f0e84034810 0x7f0e84036cc0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:51.519 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:50.958+0000 7f0ea25fb640 1 --2- 172.21.15.1:0/627400669 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0e9c101d00 0x7f0e9c110ee0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:51.519 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:50.958+0000 7f0ea25fb640 1 -- 172.21.15.1:0/627400669 >> 172.21.15.1:0/627400669 conn(0x7f0e9c0fd500 msgr2=0x7f0e9c0fe230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:30:51.519 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:50.958+0000 7f0ea25fb640 1 -- 172.21.15.1:0/627400669 shutdown_connections 2024-06-18T02:30:51.519 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:50.958+0000 7f0ea25fb640 1 -- 172.21.15.1:0/627400669 wait complete. 2024-06-18T02:30:52.377 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:52 smithi001 ceph-mon[29099]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:30:52.377 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:52 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/3140137637' entity='client.admin' 2024-06-18T02:30:52.747 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:51.919+0000 7f9f1b078640 1 Processor -- start 2024-06-18T02:30:52.747 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:51.919+0000 7f9f1b078640 1 -- start start 2024-06-18T02:30:52.747 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:51.919+0000 7f9f1b078640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f14101260 0x7f9f14101670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:52.747 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:51.919+0000 7f9f1b078640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f14101bb0 con 0x7f9f14101260 2024-06-18T02:30:52.747 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:51.919+0000 7f9f18e14640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f14101260 0x7f9f14101670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:30:52.747 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:51.919+0000 7f9f18e14640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f14101260 0x7f9f14101670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56852/0 (socket says 172.21.15.1:56852) 2024-06-18T02:30:52.747 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:51.919+0000 7f9f18e14640 1 -- 172.21.15.1:0/3096072160 learned_addr learned my addr 172.21.15.1:0/3096072160 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:30:52.748 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:51.919+0000 7f9f18e14640 1 -- 172.21.15.1:0/3096072160 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f14101cf0 con 0x7f9f14101260 2024-06-18T02:30:52.748 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:51.920+0000 7f9f18e14640 1 --2- 172.21.15.1:0/3096072160 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f14101260 0x7f9f14101670 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9f08009840 tx=0x7f9f080091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a843900bb6403f9c server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:30:52.748 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:51.920+0000 7f9f137fe640 1 -- 172.21.15.1:0/3096072160 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9f08032440 con 0x7f9f14101260 2024-06-18T02:30:52.748 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:51.920+0000 7f9f137fe640 1 -- 172.21.15.1:0/3096072160 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9f08032a70 con 0x7f9f14101260 2024-06-18T02:30:52.748 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:51.920+0000 7f9f137fe640 1 -- 172.21.15.1:0/3096072160 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9f08031ad0 con 0x7f9f14101260 2024-06-18T02:30:52.748 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:51.921+0000 7f9f1b078640 1 -- 172.21.15.1:0/3096072160 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f14101260 msgr2=0x7f9f14101670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:30:52.748 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:51.921+0000 7f9f1b078640 1 --2- 172.21.15.1:0/3096072160 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f14101260 0x7f9f14101670 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9f08009840 tx=0x7f9f080091d0 comp rx=0 tx=0).stop 2024-06-18T02:30:52.748 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:51.921+0000 7f9f1b078640 1 -- 172.21.15.1:0/3096072160 shutdown_connections 2024-06-18T02:30:52.748 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:51.921+0000 7f9f1b078640 1 --2- 172.21.15.1:0/3096072160 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f14101260 0x7f9f14101670 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:52.748 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:51.921+0000 7f9f1b078640 1 -- 172.21.15.1:0/3096072160 >> 172.21.15.1:0/3096072160 conn(0x7f9f140fbc10 msgr2=0x7f9f140fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:30:52.748 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:51.921+0000 7f9f1b078640 1 -- 172.21.15.1:0/3096072160 shutdown_connections 2024-06-18T02:30:52.748 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:51.921+0000 7f9f1b078640 1 -- 172.21.15.1:0/3096072160 wait complete. 2024-06-18T02:30:52.749 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:51.921+0000 7f9f1b078640 1 Processor -- start 2024-06-18T02:30:52.749 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:51.921+0000 7f9f1b078640 1 -- start start 2024-06-18T02:30:52.749 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:51.922+0000 7f9f1b078640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f14101260 0x7f9f14113130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:52.749 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:51.922+0000 7f9f1b078640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f14113670 con 0x7f9f14101260 2024-06-18T02:30:52.749 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:51.922+0000 7f9f18e14640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f14101260 0x7f9f14113130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:30:52.749 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:51.922+0000 7f9f18e14640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f14101260 0x7f9f14113130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56854/0 (socket says 172.21.15.1:56854) 2024-06-18T02:30:52.749 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:51.922+0000 7f9f18e14640 1 -- 172.21.15.1:0/3140137637 learned_addr learned my addr 172.21.15.1:0/3140137637 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:30:52.749 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:51.922+0000 7f9f18e14640 1 -- 172.21.15.1:0/3140137637 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f08009450 con 0x7f9f14101260 2024-06-18T02:30:52.749 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:51.922+0000 7f9f18e14640 1 --2- 172.21.15.1:0/3140137637 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f14101260 0x7f9f14113130 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9f08006a70 tx=0x7f9f0802c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:30:52.749 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:51.922+0000 7f9f117fa640 1 -- 172.21.15.1:0/3140137637 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9f08031e90 con 0x7f9f14101260 2024-06-18T02:30:52.749 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:51.922+0000 7f9f117fa640 1 -- 172.21.15.1:0/3140137637 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9f0803b910 con 0x7f9f14101260 2024-06-18T02:30:52.749 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:51.922+0000 7f9f1b078640 1 -- 172.21.15.1:0/3140137637 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f14113870 con 0x7f9f14101260 2024-06-18T02:30:52.749 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:51.922+0000 7f9f117fa640 1 -- 172.21.15.1:0/3140137637 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9f0803a940 con 0x7f9f14101260 2024-06-18T02:30:52.750 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:51.922+0000 7f9f1b078640 1 -- 172.21.15.1:0/3140137637 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f14113ce0 con 0x7f9f14101260 2024-06-18T02:30:52.750 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:51.923+0000 7f9f117fa640 1 -- 172.21.15.1:0/3140137637 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f9f08039070 con 0x7f9f14101260 2024-06-18T02:30:52.750 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:51.923+0000 7f9f117fa640 1 --2- 172.21.15.1:0/3140137637 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f9f04034810 0x7f9f04036cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:52.750 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:51.923+0000 7f9f117fa640 1 -- 172.21.15.1:0/3140137637 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f9f08066f30 con 0x7f9f14101260 2024-06-18T02:30:52.750 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:51.923+0000 7f9f1b078640 1 -- 172.21.15.1:0/3140137637 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f1405ead0 con 0x7f9f14101260 2024-06-18T02:30:52.750 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:51.924+0000 7f9f13fff640 1 --2- 172.21.15.1:0/3140137637 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f9f04034810 0x7f9f04036cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:30:52.750 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:51.924+0000 7f9f13fff640 1 --2- 172.21.15.1:0/3140137637 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f9f04034810 0x7f9f04036cc0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f9f00006fd0 tx=0x7f9f00006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:30:52.750 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:51.928+0000 7f9f117fa640 1 -- 172.21.15.1:0/3140137637 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f9f08037070 con 0x7f9f14101260 2024-06-18T02:30:52.750 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:52.102+0000 7f9f1b078640 1 -- 172.21.15.1:0/3140137637 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f9f14102c50 con 0x7f9f14101260 2024-06-18T02:30:52.750 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:52.105+0000 7f9f117fa640 1 -- 172.21.15.1:0/3140137637 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f9f08042360 con 0x7f9f14101260 2024-06-18T02:30:52.750 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:52.107+0000 7f9f1b078640 1 -- 172.21.15.1:0/3140137637 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f9f04034810 msgr2=0x7f9f04036cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:30:52.750 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:52.107+0000 7f9f1b078640 1 --2- 172.21.15.1:0/3140137637 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f9f04034810 0x7f9f04036cc0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f9f00006fd0 tx=0x7f9f00006db0 comp rx=0 tx=0).stop 2024-06-18T02:30:52.751 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:52.107+0000 7f9f1b078640 1 -- 172.21.15.1:0/3140137637 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f14101260 msgr2=0x7f9f14113130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:30:52.751 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:52.107+0000 7f9f1b078640 1 --2- 172.21.15.1:0/3140137637 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f14101260 0x7f9f14113130 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9f08006a70 tx=0x7f9f0802c040 comp rx=0 tx=0).stop 2024-06-18T02:30:52.751 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:52.107+0000 7f9f1b078640 1 -- 172.21.15.1:0/3140137637 shutdown_connections 2024-06-18T02:30:52.751 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:52.107+0000 7f9f1b078640 1 --2- 172.21.15.1:0/3140137637 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f9f04034810 0x7f9f04036cc0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:52.751 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:52.107+0000 7f9f1b078640 1 --2- 172.21.15.1:0/3140137637 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f14101260 0x7f9f14113130 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:52.751 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:52.107+0000 7f9f1b078640 1 -- 172.21.15.1:0/3140137637 >> 172.21.15.1:0/3140137637 conn(0x7f9f140fbc10 msgr2=0x7f9f140fcb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:30:52.751 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:52.107+0000 7f9f1b078640 1 -- 172.21.15.1:0/3140137637 shutdown_connections 2024-06-18T02:30:52.751 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:52.107+0000 7f9f1b078640 1 -- 172.21.15.1:0/3140137637 wait complete. 2024-06-18T02:30:52.751 INFO:teuthology.orchestra.run.smithi001.stdout:Enabling the dashboard module... 2024-06-18T02:30:54.512 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:54 smithi001 ceph-mon[29099]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:30:54.512 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:54 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/2242119692' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-06-18T02:30:54.827 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:53.133+0000 7fe8d71be640 1 Processor -- start 2024-06-18T02:30:54.827 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:53.133+0000 7fe8d71be640 1 -- start start 2024-06-18T02:30:54.827 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:53.134+0000 7fe8d71be640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8d0100c90 0x7fe8d01010a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:54.827 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:53.134+0000 7fe8d71be640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8d01015e0 con 0x7fe8d0100c90 2024-06-18T02:30:54.827 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:53.134+0000 7fe8d4f5a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8d0100c90 0x7fe8d01010a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:30:54.827 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:53.134+0000 7fe8d4f5a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8d0100c90 0x7fe8d01010a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56856/0 (socket says 172.21.15.1:56856) 2024-06-18T02:30:54.828 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:53.134+0000 7fe8d4f5a640 1 -- 172.21.15.1:0/2545086015 learned_addr learned my addr 172.21.15.1:0/2545086015 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:30:54.828 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:53.134+0000 7fe8d4f5a640 1 -- 172.21.15.1:0/2545086015 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8d0101720 con 0x7fe8d0100c90 2024-06-18T02:30:54.828 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:53.135+0000 7fe8d4f5a640 1 --2- 172.21.15.1:0/2545086015 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8d0100c90 0x7fe8d01010a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe8cc009840 tx=0x7fe8cc0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e91724f238bd59e server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:30:54.828 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:53.135+0000 7fe8c77fe640 1 -- 172.21.15.1:0/2545086015 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe8cc032440 con 0x7fe8d0100c90 2024-06-18T02:30:54.828 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:53.135+0000 7fe8c77fe640 1 -- 172.21.15.1:0/2545086015 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe8cc032a70 con 0x7fe8d0100c90 2024-06-18T02:30:54.828 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:53.135+0000 7fe8c77fe640 1 -- 172.21.15.1:0/2545086015 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe8cc031ad0 con 0x7fe8d0100c90 2024-06-18T02:30:54.828 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:53.136+0000 7fe8d71be640 1 -- 172.21.15.1:0/2545086015 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8d0100c90 msgr2=0x7fe8d01010a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:30:54.828 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:53.136+0000 7fe8d71be640 1 --2- 172.21.15.1:0/2545086015 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8d0100c90 0x7fe8d01010a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe8cc009840 tx=0x7fe8cc0091d0 comp rx=0 tx=0).stop 2024-06-18T02:30:54.828 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:53.136+0000 7fe8d71be640 1 -- 172.21.15.1:0/2545086015 shutdown_connections 2024-06-18T02:30:54.828 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:53.136+0000 7fe8d71be640 1 --2- 172.21.15.1:0/2545086015 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8d0100c90 0x7fe8d01010a0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:54.828 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:53.136+0000 7fe8d71be640 1 -- 172.21.15.1:0/2545086015 >> 172.21.15.1:0/2545086015 conn(0x7fe8d00fc450 msgr2=0x7fe8d00fe880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:30:54.828 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:53.136+0000 7fe8d71be640 1 -- 172.21.15.1:0/2545086015 shutdown_connections 2024-06-18T02:30:54.828 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:53.136+0000 7fe8d71be640 1 -- 172.21.15.1:0/2545086015 wait complete. 2024-06-18T02:30:54.828 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:53.136+0000 7fe8d71be640 1 Processor -- start 2024-06-18T02:30:54.829 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:53.137+0000 7fe8d71be640 1 -- start start 2024-06-18T02:30:54.829 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:53.137+0000 7fe8d71be640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8d0100c90 0x7fe8d0075b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:54.829 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:53.137+0000 7fe8d71be640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8d01138e0 con 0x7fe8d0100c90 2024-06-18T02:30:54.829 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:53.137+0000 7fe8d4f5a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8d0100c90 0x7fe8d0075b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:30:54.829 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:53.137+0000 7fe8d4f5a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8d0100c90 0x7fe8d0075b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56866/0 (socket says 172.21.15.1:56866) 2024-06-18T02:30:54.829 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:53.137+0000 7fe8d4f5a640 1 -- 172.21.15.1:0/2242119692 learned_addr learned my addr 172.21.15.1:0/2242119692 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:30:54.829 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:53.137+0000 7fe8d4f5a640 1 -- 172.21.15.1:0/2242119692 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8cc009450 con 0x7fe8d0100c90 2024-06-18T02:30:54.829 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:53.138+0000 7fe8d4f5a640 1 --2- 172.21.15.1:0/2242119692 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8d0100c90 0x7fe8d0075b20 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe8cc009810 tx=0x7fe8cc02c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:30:54.829 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:53.138+0000 7fe8c57fa640 1 -- 172.21.15.1:0/2242119692 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe8cc0328b0 con 0x7fe8d0100c90 2024-06-18T02:30:54.829 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:53.138+0000 7fe8d71be640 1 -- 172.21.15.1:0/2242119692 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8d0113a20 con 0x7fe8d0100c90 2024-06-18T02:30:54.829 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:53.138+0000 7fe8c57fa640 1 -- 172.21.15.1:0/2242119692 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe8cc03b910 con 0x7fe8d0100c90 2024-06-18T02:30:54.829 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:53.138+0000 7fe8c57fa640 1 -- 172.21.15.1:0/2242119692 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe8cc03a980 con 0x7fe8d0100c90 2024-06-18T02:30:54.829 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:53.138+0000 7fe8d71be640 1 -- 172.21.15.1:0/2242119692 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8d0113e30 con 0x7fe8d0100c90 2024-06-18T02:30:54.829 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:53.139+0000 7fe8c57fa640 1 -- 172.21.15.1:0/2242119692 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7fe8cc039070 con 0x7fe8d0100c90 2024-06-18T02:30:54.829 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:53.139+0000 7fe8c57fa640 1 --2- 172.21.15.1:0/2242119692 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7fe8b8034810 0x7fe8b8036cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:54.830 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:53.139+0000 7fe8c57fa640 1 -- 172.21.15.1:0/2242119692 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fe8cc067060 con 0x7fe8d0100c90 2024-06-18T02:30:54.830 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:53.139+0000 7fe8d71be640 1 -- 172.21.15.1:0/2242119692 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8d004b480 con 0x7fe8d0100c90 2024-06-18T02:30:54.830 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:53.139+0000 7fe8c7fff640 1 --2- 172.21.15.1:0/2242119692 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7fe8b8034810 0x7fe8b8036cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:30:54.830 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:53.140+0000 7fe8c7fff640 1 --2- 172.21.15.1:0/2242119692 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7fe8b8034810 0x7fe8b8036cc0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe8c0006fd0 tx=0x7fe8c0006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:30:54.830 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:53.144+0000 7fe8c57fa640 1 -- 172.21.15.1:0/2242119692 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fe8cc037070 con 0x7fe8d0100c90 2024-06-18T02:30:54.830 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:53.362+0000 7fe8d71be640 1 -- 172.21.15.1:0/2242119692 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7fe8d0102a20 con 0x7fe8d0100c90 2024-06-18T02:30:54.830 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:54.248+0000 7fe8c57fa640 1 -- 172.21.15.1:0/2242119692 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 9) v1 ==== 41959+0+0 (secure 0 0 0) 0x7fe8cc04c350 con 0x7fe8d0100c90 2024-06-18T02:30:54.830 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:54.250+0000 7fe8c57fa640 1 -- 172.21.15.1:0/2242119692 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7fe8cc03aba0 con 0x7fe8d0100c90 2024-06-18T02:30:54.830 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:54.253+0000 7fe8d71be640 1 -- 172.21.15.1:0/2242119692 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7fe8b8034810 msgr2=0x7fe8b8036cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:30:54.830 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:54.253+0000 7fe8d71be640 1 --2- 172.21.15.1:0/2242119692 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7fe8b8034810 0x7fe8b8036cc0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe8c0006fd0 tx=0x7fe8c0006db0 comp rx=0 tx=0).stop 2024-06-18T02:30:54.830 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:54.253+0000 7fe8d71be640 1 -- 172.21.15.1:0/2242119692 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8d0100c90 msgr2=0x7fe8d0075b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:30:54.830 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:54.253+0000 7fe8d71be640 1 --2- 172.21.15.1:0/2242119692 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8d0100c90 0x7fe8d0075b20 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe8cc009810 tx=0x7fe8cc02c040 comp rx=0 tx=0).stop 2024-06-18T02:30:54.831 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:54.253+0000 7fe8d71be640 1 -- 172.21.15.1:0/2242119692 shutdown_connections 2024-06-18T02:30:54.831 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:54.253+0000 7fe8d71be640 1 --2- 172.21.15.1:0/2242119692 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7fe8b8034810 0x7fe8b8036cc0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:54.831 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:54.253+0000 7fe8d71be640 1 --2- 172.21.15.1:0/2242119692 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe8d0100c90 0x7fe8d0075b20 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:54.831 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:54.253+0000 7fe8d71be640 1 -- 172.21.15.1:0/2242119692 >> 172.21.15.1:0/2242119692 conn(0x7fe8d00fc450 msgr2=0x7fe8d00fd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:30:54.831 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:54.253+0000 7fe8d71be640 1 -- 172.21.15.1:0/2242119692 shutdown_connections 2024-06-18T02:30:54.831 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:54.253+0000 7fe8d71be640 1 -- 172.21.15.1:0/2242119692 wait complete. 2024-06-18T02:30:55.432 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:55 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/2242119692' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-06-18T02:30:55.432 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:55 smithi001 ceph-mon[29099]: mgrmap e9: smithi001.teoawz(active, since 25s) 2024-06-18T02:30:56.068 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout { 2024-06-18T02:30:56.069 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-06-18T02:30:56.069 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "available": true, 2024-06-18T02:30:56.069 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "active_name": "smithi001.teoawz", 2024-06-18T02:30:56.069 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-06-18T02:30:56.069 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout } 2024-06-18T02:30:56.069 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:55.226+0000 7fd16c9a5640 1 Processor -- start 2024-06-18T02:30:56.069 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:55.226+0000 7fd16c9a5640 1 -- start start 2024-06-18T02:30:56.069 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:55.227+0000 7fd16c9a5640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd168075aa0 0x7fd168104a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:56.069 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:55.227+0000 7fd16c9a5640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd168075eb0 con 0x7fd168075aa0 2024-06-18T02:30:56.070 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:55.227+0000 7fd16659c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd168075aa0 0x7fd168104a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:30:56.070 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:55.227+0000 7fd16659c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd168075aa0 0x7fd168104a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42516/0 (socket says 172.21.15.1:42516) 2024-06-18T02:30:56.070 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:55.227+0000 7fd16659c640 1 -- 172.21.15.1:0/3605364334 learned_addr learned my addr 172.21.15.1:0/3605364334 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:30:56.070 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:55.227+0000 7fd16659c640 1 -- 172.21.15.1:0/3605364334 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd168104fa0 con 0x7fd168075aa0 2024-06-18T02:30:56.070 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:55.228+0000 7fd16659c640 1 --2- 172.21.15.1:0/3605364334 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd168075aa0 0x7fd168104a60 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd158009840 tx=0x7fd1580091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86909a44e4bffadf server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:30:56.070 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:55.228+0000 7fd165d9b640 1 -- 172.21.15.1:0/3605364334 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd158032440 con 0x7fd168075aa0 2024-06-18T02:30:56.070 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:55.228+0000 7fd165d9b640 1 -- 172.21.15.1:0/3605364334 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd158032a70 con 0x7fd168075aa0 2024-06-18T02:30:56.070 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:55.228+0000 7fd165d9b640 1 -- 172.21.15.1:0/3605364334 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd158031ad0 con 0x7fd168075aa0 2024-06-18T02:30:56.070 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:55.229+0000 7fd16c9a5640 1 -- 172.21.15.1:0/3605364334 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd168075aa0 msgr2=0x7fd168104a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:30:56.070 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:55.229+0000 7fd16c9a5640 1 --2- 172.21.15.1:0/3605364334 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd168075aa0 0x7fd168104a60 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd158009840 tx=0x7fd1580091d0 comp rx=0 tx=0).stop 2024-06-18T02:30:56.070 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:55.229+0000 7fd16c9a5640 1 -- 172.21.15.1:0/3605364334 shutdown_connections 2024-06-18T02:30:56.070 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:55.229+0000 7fd16c9a5640 1 --2- 172.21.15.1:0/3605364334 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd168075aa0 0x7fd168104a60 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:56.071 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:55.229+0000 7fd16c9a5640 1 -- 172.21.15.1:0/3605364334 >> 172.21.15.1:0/3605364334 conn(0x7fd1680fc450 msgr2=0x7fd1680fe880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:30:56.071 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:55.229+0000 7fd16c9a5640 1 -- 172.21.15.1:0/3605364334 shutdown_connections 2024-06-18T02:30:56.071 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:55.229+0000 7fd16c9a5640 1 -- 172.21.15.1:0/3605364334 wait complete. 2024-06-18T02:30:56.071 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:55.230+0000 7fd16c9a5640 1 Processor -- start 2024-06-18T02:30:56.071 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:55.230+0000 7fd16c9a5640 1 -- start start 2024-06-18T02:30:56.071 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:55.230+0000 7fd16c9a5640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd168075aa0 0x7fd168110bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:56.071 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:55.230+0000 7fd16c9a5640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd168111100 con 0x7fd168075aa0 2024-06-18T02:30:56.071 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:55.230+0000 7fd16659c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd168075aa0 0x7fd168110bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:30:56.071 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:55.230+0000 7fd16659c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd168075aa0 0x7fd168110bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42532/0 (socket says 172.21.15.1:42532) 2024-06-18T02:30:56.072 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:55.230+0000 7fd16659c640 1 -- 172.21.15.1:0/3530628226 learned_addr learned my addr 172.21.15.1:0/3530628226 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:30:56.072 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:55.230+0000 7fd16659c640 1 -- 172.21.15.1:0/3530628226 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd158009450 con 0x7fd168075aa0 2024-06-18T02:30:56.072 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:55.231+0000 7fd16659c640 1 --2- 172.21.15.1:0/3530628226 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd168075aa0 0x7fd168110bc0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd1580098a0 tx=0x7fd15802c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:30:56.072 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:55.231+0000 7fd15effd640 1 -- 172.21.15.1:0/3530628226 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd158031e90 con 0x7fd168075aa0 2024-06-18T02:30:56.072 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:55.231+0000 7fd15effd640 1 -- 172.21.15.1:0/3530628226 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd15803b910 con 0x7fd168075aa0 2024-06-18T02:30:56.072 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:55.231+0000 7fd16c9a5640 1 -- 172.21.15.1:0/3530628226 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd168111300 con 0x7fd168075aa0 2024-06-18T02:30:56.072 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:55.231+0000 7fd15effd640 1 -- 172.21.15.1:0/3530628226 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd15803a9b0 con 0x7fd168075aa0 2024-06-18T02:30:56.072 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:55.231+0000 7fd16c9a5640 1 -- 172.21.15.1:0/3530628226 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd168111770 con 0x7fd168075aa0 2024-06-18T02:30:56.072 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:55.232+0000 7fd15effd640 1 -- 172.21.15.1:0/3530628226 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 9) v1 ==== 41959+0+0 (secure 0 0 0) 0x7fd158039070 con 0x7fd168075aa0 2024-06-18T02:30:56.072 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:55.232+0000 7fd15effd640 1 --2- 172.21.15.1:0/3530628226 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7fd154034860 0x7fd154036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:30:56.072 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:55.232+0000 7fd16c9a5640 1 -- 172.21.15.1:0/3530628226 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd16804b480 con 0x7fd168075aa0 2024-06-18T02:30:56.072 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:55.232+0000 7fd15ffff640 1 -- 172.21.15.1:0/3530628226 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7fd154034860 msgr2=0x7fd154036d10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/4268462134 2024-06-18T02:30:56.073 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:55.232+0000 7fd15effd640 1 -- 172.21.15.1:0/3530628226 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fd158066e50 con 0x7fd168075aa0 2024-06-18T02:30:56.073 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:55.232+0000 7fd15ffff640 1 --2- 172.21.15.1:0/3530628226 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7fd154034860 0x7fd154036d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-18T02:30:56.073 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:55.237+0000 7fd15effd640 1 -- 172.21.15.1:0/3530628226 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fd158039910 con 0x7fd168075aa0 2024-06-18T02:30:56.073 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:55.433+0000 7fd15ffff640 1 -- 172.21.15.1:0/3530628226 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7fd154034860 msgr2=0x7fd154036d10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/4268462134 2024-06-18T02:30:56.073 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:55.433+0000 7fd15ffff640 1 --2- 172.21.15.1:0/3530628226 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7fd154034860 0x7fd154036d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-18T02:30:56.073 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:55.459+0000 7fd16c9a5640 1 -- 172.21.15.1:0/3530628226 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fd168105d30 con 0x7fd168075aa0 2024-06-18T02:30:56.073 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:55.459+0000 7fd15effd640 1 -- 172.21.15.1:0/3530628226 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+103 (secure 0 0 0) 0x7fd158037070 con 0x7fd168075aa0 2024-06-18T02:30:56.073 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:55.461+0000 7fd16c9a5640 1 -- 172.21.15.1:0/3530628226 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7fd154034860 msgr2=0x7fd154036d10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-18T02:30:56.073 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:55.461+0000 7fd16c9a5640 1 --2- 172.21.15.1:0/3530628226 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7fd154034860 0x7fd154036d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:56.073 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:55.461+0000 7fd16c9a5640 1 -- 172.21.15.1:0/3530628226 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd168075aa0 msgr2=0x7fd168110bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:30:56.073 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:55.461+0000 7fd16c9a5640 1 --2- 172.21.15.1:0/3530628226 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd168075aa0 0x7fd168110bc0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd1580098a0 tx=0x7fd15802c040 comp rx=0 tx=0).stop 2024-06-18T02:30:56.073 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:55.461+0000 7fd16c9a5640 1 -- 172.21.15.1:0/3530628226 shutdown_connections 2024-06-18T02:30:56.073 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:55.461+0000 7fd16c9a5640 1 --2- 172.21.15.1:0/3530628226 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7fd154034860 0x7fd154036d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:56.073 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:55.461+0000 7fd16c9a5640 1 --2- 172.21.15.1:0/3530628226 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd168075aa0 0x7fd168110bc0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:30:56.074 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:55.461+0000 7fd16c9a5640 1 -- 172.21.15.1:0/3530628226 >> 172.21.15.1:0/3530628226 conn(0x7fd1680fc450 msgr2=0x7fd1680fd120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:30:56.074 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:55.461+0000 7fd16c9a5640 1 -- 172.21.15.1:0/3530628226 shutdown_connections 2024-06-18T02:30:56.074 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:55.461+0000 7fd16c9a5640 1 -- 172.21.15.1:0/3530628226 wait complete. 2024-06-18T02:30:56.074 INFO:teuthology.orchestra.run.smithi001.stdout:Waiting for the mgr to restart... 2024-06-18T02:30:56.074 INFO:teuthology.orchestra.run.smithi001.stdout:Waiting for mgr epoch 9... 2024-06-18T02:30:56.297 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:56 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/3530628226' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-06-18T02:30:59.127 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:59 smithi001 ceph-mon[29099]: Active manager daemon smithi001.teoawz restarted 2024-06-18T02:30:59.127 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:59 smithi001 ceph-mon[29099]: Activating manager daemon smithi001.teoawz 2024-06-18T02:30:59.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:59 smithi001 ceph-mon[29099]: osdmap e3: 0 total, 0 up, 0 in 2024-06-18T02:30:59.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:59 smithi001 ceph-mon[29099]: mgrmap e10: smithi001.teoawz(active, starting, since 0.00307862s) 2024-06-18T02:30:59.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:59 smithi001 ceph-mon[29099]: from='mgr.14160 172.21.15.1:0/1498115122' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "mon metadata", "id": "smithi001"}]: dispatch 2024-06-18T02:30:59.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:59 smithi001 ceph-mon[29099]: from='mgr.14160 172.21.15.1:0/1498115122' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "mgr metadata", "who": "smithi001.teoawz", "id": "smithi001.teoawz"}]: dispatch 2024-06-18T02:30:59.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:59 smithi001 ceph-mon[29099]: from='mgr.14160 172.21.15.1:0/1498115122' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-18T02:30:59.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:59 smithi001 ceph-mon[29099]: from='mgr.14160 172.21.15.1:0/1498115122' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-18T02:30:59.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:59 smithi001 ceph-mon[29099]: from='mgr.14160 172.21.15.1:0/1498115122' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-18T02:30:59.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:59 smithi001 ceph-mon[29099]: Manager daemon smithi001.teoawz is now available 2024-06-18T02:30:59.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:59 smithi001 ceph-mon[29099]: from='mgr.14160 172.21.15.1:0/1498115122' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:30:59.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:59 smithi001 ceph-mon[29099]: from='mgr.14160 172.21.15.1:0/1498115122' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:30:59.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:59 smithi001 ceph-mon[29099]: from='mgr.14160 172.21.15.1:0/1498115122' entity='mgr.smithi001.teoawz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi001.teoawz/mirror_snapshot_schedule"}]: dispatch 2024-06-18T02:30:59.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:30:59 smithi001 ceph-mon[29099]: from='mgr.14160 172.21.15.1:0/1498115122' entity='mgr.smithi001.teoawz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi001.teoawz/trash_purge_schedule"}]: dispatch 2024-06-18T02:31:00.863 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout { 2024-06-18T02:31:00.863 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-06-18T02:31:00.863 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout "initialized": true 2024-06-18T02:31:00.863 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout } 2024-06-18T02:31:00.863 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:56.542+0000 7f34d5d05640 1 Processor -- start 2024-06-18T02:31:00.863 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:56.542+0000 7f34d5d05640 1 -- start start 2024-06-18T02:31:00.863 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:56.542+0000 7f34d5d05640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34d0075b20 0x7f34d0106dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:31:00.864 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:56.542+0000 7f34d5d05640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34d0075f30 con 0x7f34d0075b20 2024-06-18T02:31:00.864 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:56.543+0000 7f34cf7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34d0075b20 0x7f34d0106dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:31:00.864 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:56.543+0000 7f34cf7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34d0075b20 0x7f34d0106dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42534/0 (socket says 172.21.15.1:42534) 2024-06-18T02:31:00.864 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:56.543+0000 7f34cf7fe640 1 -- 172.21.15.1:0/986055873 learned_addr learned my addr 172.21.15.1:0/986055873 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:31:00.864 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:56.543+0000 7f34cf7fe640 1 -- 172.21.15.1:0/986055873 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34d01027c0 con 0x7f34d0075b20 2024-06-18T02:31:00.864 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:56.544+0000 7f34cf7fe640 1 --2- 172.21.15.1:0/986055873 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34d0075b20 0x7f34d0106dd0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f34c0009840 tx=0x7f34c00091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3efff4b730376777 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:31:00.864 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:56.544+0000 7f34ce7fc640 1 -- 172.21.15.1:0/986055873 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34c0032440 con 0x7f34d0075b20 2024-06-18T02:31:00.864 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:56.544+0000 7f34ce7fc640 1 -- 172.21.15.1:0/986055873 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f34c0032a70 con 0x7f34d0075b20 2024-06-18T02:31:00.864 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:56.544+0000 7f34ce7fc640 1 -- 172.21.15.1:0/986055873 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34c0031ad0 con 0x7f34d0075b20 2024-06-18T02:31:00.864 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:56.545+0000 7f34d5d05640 1 -- 172.21.15.1:0/986055873 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34d0075b20 msgr2=0x7f34d0106dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:31:00.864 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:56.545+0000 7f34d5d05640 1 --2- 172.21.15.1:0/986055873 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34d0075b20 0x7f34d0106dd0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f34c0009840 tx=0x7f34c00091d0 comp rx=0 tx=0).stop 2024-06-18T02:31:00.864 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:56.545+0000 7f34d5d05640 1 -- 172.21.15.1:0/986055873 shutdown_connections 2024-06-18T02:31:00.864 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:56.545+0000 7f34d5d05640 1 --2- 172.21.15.1:0/986055873 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34d0075b20 0x7f34d0106dd0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:31:00.864 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:56.545+0000 7f34d5d05640 1 -- 172.21.15.1:0/986055873 >> 172.21.15.1:0/986055873 conn(0x7f34d00fc460 msgr2=0x7f34d00fe870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:31:00.865 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:56.545+0000 7f34d5d05640 1 -- 172.21.15.1:0/986055873 shutdown_connections 2024-06-18T02:31:00.865 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:56.545+0000 7f34d5d05640 1 -- 172.21.15.1:0/986055873 wait complete. 2024-06-18T02:31:00.865 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:56.546+0000 7f34d5d05640 1 Processor -- start 2024-06-18T02:31:00.865 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:56.546+0000 7f34d5d05640 1 -- start start 2024-06-18T02:31:00.865 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:56.546+0000 7f34d5d05640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34d0075b20 0x7f34d0112e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:31:00.865 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:56.546+0000 7f34d5d05640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34d0113340 con 0x7f34d0075b20 2024-06-18T02:31:00.865 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:56.546+0000 7f34cf7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34d0075b20 0x7f34d0112e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:31:00.865 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:56.546+0000 7f34cf7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34d0075b20 0x7f34d0112e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42540/0 (socket says 172.21.15.1:42540) 2024-06-18T02:31:00.865 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:56.546+0000 7f34cf7fe640 1 -- 172.21.15.1:0/4004765899 learned_addr learned my addr 172.21.15.1:0/4004765899 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:31:00.865 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:56.547+0000 7f34cf7fe640 1 -- 172.21.15.1:0/4004765899 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34c0009450 con 0x7f34d0075b20 2024-06-18T02:31:00.865 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:56.547+0000 7f34cf7fe640 1 --2- 172.21.15.1:0/4004765899 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34d0075b20 0x7f34d0112e00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f34c00317f0 tx=0x7f34c002c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:31:00.865 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:56.547+0000 7f34b7fff640 1 -- 172.21.15.1:0/4004765899 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34c0032900 con 0x7f34d0075b20 2024-06-18T02:31:00.865 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:56.547+0000 7f34b7fff640 1 -- 172.21.15.1:0/4004765899 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f34c003b910 con 0x7f34d0075b20 2024-06-18T02:31:00.865 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:56.547+0000 7f34d5d05640 1 -- 172.21.15.1:0/4004765899 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34d01135a0 con 0x7f34d0075b20 2024-06-18T02:31:00.865 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:56.547+0000 7f34b7fff640 1 -- 172.21.15.1:0/4004765899 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34c003a960 con 0x7f34d0075b20 2024-06-18T02:31:00.866 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:56.547+0000 7f34d5d05640 1 -- 172.21.15.1:0/4004765899 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34d0113f90 con 0x7f34d0075b20 2024-06-18T02:31:00.866 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:56.548+0000 7f34b7fff640 1 -- 172.21.15.1:0/4004765899 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 9) v1 ==== 41959+0+0 (secure 0 0 0) 0x7f34c0039070 con 0x7f34d0075b20 2024-06-18T02:31:00.866 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:56.548+0000 7f34b7fff640 1 --2- 172.21.15.1:0/4004765899 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f34b80348b0 0x7f34b8036d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:31:00.866 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:56.548+0000 7f34b7fff640 1 -- 172.21.15.1:0/4004765899 --> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f34c0032900 con 0x7f34b80348b0 2024-06-18T02:31:00.866 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:56.548+0000 7f34b7fff640 1 -- 172.21.15.1:0/4004765899 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f34c0066de0 con 0x7f34d0075b20 2024-06-18T02:31:00.866 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:56.548+0000 7f34ceffd640 1 -- 172.21.15.1:0/4004765899 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f34b80348b0 msgr2=0x7f34b8036d60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/4268462134 2024-06-18T02:31:00.866 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:56.548+0000 7f34ceffd640 1 --2- 172.21.15.1:0/4004765899 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f34b80348b0 0x7f34b8036d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-18T02:31:00.866 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:56.749+0000 7f34ceffd640 1 -- 172.21.15.1:0/4004765899 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f34b80348b0 msgr2=0x7f34b8036d60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/4268462134 2024-06-18T02:31:00.866 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:56.749+0000 7f34ceffd640 1 --2- 172.21.15.1:0/4004765899 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f34b80348b0 0x7f34b8036d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-18T02:31:00.866 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:57.150+0000 7f34ceffd640 1 -- 172.21.15.1:0/4004765899 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f34b80348b0 msgr2=0x7f34b8036d60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/4268462134 2024-06-18T02:31:00.866 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:57.150+0000 7f34ceffd640 1 --2- 172.21.15.1:0/4004765899 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f34b80348b0 0x7f34b8036d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-18T02:31:00.866 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:57.951+0000 7f34ceffd640 1 -- 172.21.15.1:0/4004765899 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f34b80348b0 msgr2=0x7f34b8036d60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/4268462134 2024-06-18T02:31:00.867 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:57.951+0000 7f34ceffd640 1 --2- 172.21.15.1:0/4004765899 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f34b80348b0 0x7f34b8036d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-18T02:31:00.867 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:58.991+0000 7f34b7fff640 1 -- 172.21.15.1:0/4004765899 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mgrmap(e 10) v1 ==== 41726+0+0 (secure 0 0 0) 0x7f34c0065f80 con 0x7f34d0075b20 2024-06-18T02:31:00.867 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:58.991+0000 7f34b7fff640 1 -- 172.21.15.1:0/4004765899 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f34b80348b0 msgr2=0x7f34b8036d60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-18T02:31:00.867 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:58.991+0000 7f34b7fff640 1 --2- 172.21.15.1:0/4004765899 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f34b80348b0 0x7f34b8036d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:31:00.867 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:59.992+0000 7f34b7fff640 1 -- 172.21.15.1:0/4004765899 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 11) v1 ==== 41853+0+0 (secure 0 0 0) 0x7f34c004c350 con 0x7f34d0075b20 2024-06-18T02:31:00.867 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:59.992+0000 7f34b7fff640 1 --2- 172.21.15.1:0/4004765899 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f34b8037fd0 0x7f34b803a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:31:00.867 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:59.992+0000 7f34b7fff640 1 -- 172.21.15.1:0/4004765899 --> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f34c0032900 con 0x7f34b8037fd0 2024-06-18T02:31:00.867 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:59.993+0000 7f34ceffd640 1 --2- 172.21.15.1:0/4004765899 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f34b8037fd0 0x7f34b803a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:31:00.867 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:59.994+0000 7f34ceffd640 1 --2- 172.21.15.1:0/4004765899 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f34b8037fd0 0x7f34b803a3b0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f34c4003940 tx=0x7f34c4009350 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:31:00.867 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:30:59.994+0000 7f34b7fff640 1 -- 172.21.15.1:0/4004765899 <== mgr.14160 v2:172.21.15.1:6800/85550937 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7f34c0032900 con 0x7f34b8037fd0 2024-06-18T02:31:00.867 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:00.004+0000 7f34d5d05640 1 -- 172.21.15.1:0/4004765899 --> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f34d005ead0 con 0x7f34b8037fd0 2024-06-18T02:31:00.867 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:00.004+0000 7f34b7fff640 1 -- 172.21.15.1:0/4004765899 <== mgr.14160 v2:172.21.15.1:6800/85550937 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f34d005ead0 con 0x7f34b8037fd0 2024-06-18T02:31:00.867 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:00.004+0000 7f34d5d05640 1 -- 172.21.15.1:0/4004765899 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f34b8037fd0 msgr2=0x7f34b803a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:31:00.868 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:00.004+0000 7f34d5d05640 1 --2- 172.21.15.1:0/4004765899 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f34b8037fd0 0x7f34b803a3b0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f34c4003940 tx=0x7f34c4009350 comp rx=0 tx=0).stop 2024-06-18T02:31:00.868 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:00.004+0000 7f34d5d05640 1 -- 172.21.15.1:0/4004765899 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34d0075b20 msgr2=0x7f34d0112e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:31:00.868 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:00.004+0000 7f34d5d05640 1 --2- 172.21.15.1:0/4004765899 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34d0075b20 0x7f34d0112e00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f34c00317f0 tx=0x7f34c002c040 comp rx=0 tx=0).stop 2024-06-18T02:31:00.868 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:00.004+0000 7f34d5d05640 1 -- 172.21.15.1:0/4004765899 shutdown_connections 2024-06-18T02:31:00.868 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:00.004+0000 7f34d5d05640 1 --2- 172.21.15.1:0/4004765899 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f34b8037fd0 0x7f34b803a3b0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:31:00.868 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:00.004+0000 7f34d5d05640 1 --2- 172.21.15.1:0/4004765899 >> [v2:172.21.15.1:6800/4268462134,v1:172.21.15.1:6801/4268462134] conn(0x7f34b80348b0 0x7f34b8036d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:31:00.868 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:00.004+0000 7f34d5d05640 1 --2- 172.21.15.1:0/4004765899 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34d0075b20 0x7f34d0112e00 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:31:00.868 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:00.004+0000 7f34d5d05640 1 -- 172.21.15.1:0/4004765899 >> 172.21.15.1:0/4004765899 conn(0x7f34d00fc460 msgr2=0x7f34d00fd0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:31:00.868 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:00.005+0000 7f34d5d05640 1 -- 172.21.15.1:0/4004765899 shutdown_connections 2024-06-18T02:31:00.868 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:00.005+0000 7f34d5d05640 1 -- 172.21.15.1:0/4004765899 wait complete. 2024-06-18T02:31:00.868 INFO:teuthology.orchestra.run.smithi001.stdout:mgr epoch 9 is available 2024-06-18T02:31:00.868 INFO:teuthology.orchestra.run.smithi001.stdout:Generating a dashboard self-signed certificate... 2024-06-18T02:31:01.273 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:00 smithi001 ceph-mon[29099]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-06-18T02:31:01.273 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:00 smithi001 ceph-mon[29099]: mgrmap e11: smithi001.teoawz(active, since 1.00545s) 2024-06-18T02:31:01.274 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:00 smithi001 ceph-mon[29099]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-06-18T02:31:01.274 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:00 smithi001 ceph-mon[29099]: from='mgr.14160 172.21.15.1:0/1498115122' entity='mgr.smithi001.teoawz' 2024-06-18T02:31:01.274 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:00 smithi001 ceph-mon[29099]: from='mgr.14160 172.21.15.1:0/1498115122' entity='mgr.smithi001.teoawz' 2024-06-18T02:31:02.168 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-06-18T02:31:02.168 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:01.291+0000 7f4afe1e8640 1 Processor -- start 2024-06-18T02:31:02.168 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:01.291+0000 7f4afe1e8640 1 -- start start 2024-06-18T02:31:02.168 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:01.292+0000 7f4afe1e8640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4af8077c30 0x7f4af8076250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:31:02.168 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:01.292+0000 7f4afe1e8640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4af8078040 con 0x7f4af8077c30 2024-06-18T02:31:02.169 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:01.292+0000 7f4af77fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4af8077c30 0x7f4af8076250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:31:02.169 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:01.292+0000 7f4af77fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4af8077c30 0x7f4af8076250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42624/0 (socket says 172.21.15.1:42624) 2024-06-18T02:31:02.169 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:01.292+0000 7f4af77fe640 1 -- 172.21.15.1:0/1147934366 learned_addr learned my addr 172.21.15.1:0/1147934366 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:31:02.169 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:01.292+0000 7f4af77fe640 1 -- 172.21.15.1:0/1147934366 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4af8076790 con 0x7f4af8077c30 2024-06-18T02:31:02.169 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:01.293+0000 7f4af77fe640 1 --2- 172.21.15.1:0/1147934366 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4af8077c30 0x7f4af8076250 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4ae8009840 tx=0x7f4ae80091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d578efba97a5689 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:31:02.169 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:01.293+0000 7f4af67fc640 1 -- 172.21.15.1:0/1147934366 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ae8032440 con 0x7f4af8077c30 2024-06-18T02:31:02.169 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:01.293+0000 7f4af67fc640 1 -- 172.21.15.1:0/1147934366 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4ae8032a70 con 0x7f4af8077c30 2024-06-18T02:31:02.169 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:01.293+0000 7f4af67fc640 1 -- 172.21.15.1:0/1147934366 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ae8031ad0 con 0x7f4af8077c30 2024-06-18T02:31:02.169 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:01.294+0000 7f4afe1e8640 1 -- 172.21.15.1:0/1147934366 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4af8077c30 msgr2=0x7f4af8076250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:31:02.169 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:01.294+0000 7f4afe1e8640 1 --2- 172.21.15.1:0/1147934366 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4af8077c30 0x7f4af8076250 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4ae8009840 tx=0x7f4ae80091d0 comp rx=0 tx=0).stop 2024-06-18T02:31:02.169 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:01.294+0000 7f4afe1e8640 1 -- 172.21.15.1:0/1147934366 shutdown_connections 2024-06-18T02:31:02.169 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:01.294+0000 7f4afe1e8640 1 --2- 172.21.15.1:0/1147934366 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4af8077c30 0x7f4af8076250 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:31:02.169 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:01.294+0000 7f4afe1e8640 1 -- 172.21.15.1:0/1147934366 >> 172.21.15.1:0/1147934366 conn(0x7f4af80fd5e0 msgr2=0x7f4af80ff9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:31:02.169 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:01.294+0000 7f4afe1e8640 1 -- 172.21.15.1:0/1147934366 shutdown_connections 2024-06-18T02:31:02.170 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:01.294+0000 7f4afe1e8640 1 -- 172.21.15.1:0/1147934366 wait complete. 2024-06-18T02:31:02.170 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:01.295+0000 7f4afe1e8640 1 Processor -- start 2024-06-18T02:31:02.170 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:01.295+0000 7f4afe1e8640 1 -- start start 2024-06-18T02:31:02.170 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:01.295+0000 7f4afe1e8640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4af8077c30 0x7f4af8112e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:31:02.170 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:01.295+0000 7f4afe1e8640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4af8113360 con 0x7f4af8077c30 2024-06-18T02:31:02.170 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:01.295+0000 7f4af77fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4af8077c30 0x7f4af8112e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:31:02.170 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:01.295+0000 7f4af77fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4af8077c30 0x7f4af8112e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42630/0 (socket says 172.21.15.1:42630) 2024-06-18T02:31:02.170 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:01.295+0000 7f4af77fe640 1 -- 172.21.15.1:0/4107709725 learned_addr learned my addr 172.21.15.1:0/4107709725 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:31:02.170 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:01.295+0000 7f4af77fe640 1 -- 172.21.15.1:0/4107709725 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ae8009450 con 0x7f4af8077c30 2024-06-18T02:31:02.170 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:01.295+0000 7f4af77fe640 1 --2- 172.21.15.1:0/4107709725 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4af8077c30 0x7f4af8112e20 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4ae80317f0 tx=0x7f4ae802c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:31:02.170 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:01.296+0000 7f4adffff640 1 -- 172.21.15.1:0/4107709725 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ae8032900 con 0x7f4af8077c30 2024-06-18T02:31:02.170 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:01.296+0000 7f4adffff640 1 -- 172.21.15.1:0/4107709725 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4ae803b910 con 0x7f4af8077c30 2024-06-18T02:31:02.170 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:01.296+0000 7f4adffff640 1 -- 172.21.15.1:0/4107709725 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ae803a960 con 0x7f4af8077c30 2024-06-18T02:31:02.170 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:01.296+0000 7f4afe1e8640 1 -- 172.21.15.1:0/4107709725 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4af8113560 con 0x7f4af8077c30 2024-06-18T02:31:02.171 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:01.296+0000 7f4afe1e8640 1 -- 172.21.15.1:0/4107709725 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4af8114040 con 0x7f4af8077c30 2024-06-18T02:31:02.171 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:01.297+0000 7f4adffff640 1 -- 172.21.15.1:0/4107709725 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 11) v1 ==== 41853+0+0 (secure 0 0 0) 0x7f4ae8039070 con 0x7f4af8077c30 2024-06-18T02:31:02.171 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:01.297+0000 7f4adffff640 1 --2- 172.21.15.1:0/4107709725 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f4ae0034730 0x7f4ae0036be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:31:02.171 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:01.297+0000 7f4adffff640 1 -- 172.21.15.1:0/4107709725 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f4ae8066db0 con 0x7f4af8077c30 2024-06-18T02:31:02.171 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:01.297+0000 7f4af6ffd640 1 --2- 172.21.15.1:0/4107709725 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f4ae0034730 0x7f4ae0036be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:31:02.171 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:01.297+0000 7f4afe1e8640 1 -- 172.21.15.1:0/4107709725 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4af805ead0 con 0x7f4af8077c30 2024-06-18T02:31:02.171 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:01.297+0000 7f4af6ffd640 1 --2- 172.21.15.1:0/4107709725 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f4ae0034730 0x7f4ae0036be0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4aec006fd0 tx=0x7f4aec006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:31:02.171 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:01.302+0000 7f4adffff640 1 -- 172.21.15.1:0/4107709725 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f4ae8037070 con 0x7f4af8077c30 2024-06-18T02:31:02.171 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:01.483+0000 7f4afe1e8640 1 -- 172.21.15.1:0/4107709725 --> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f4af80fe9d0 con 0x7f4ae0034730 2024-06-18T02:31:02.171 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:01.535+0000 7f4adffff640 1 -- 172.21.15.1:0/4107709725 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mgrmap(e 12) v1 ==== 41959+0+0 (secure 0 0 0) 0x7f4ae804c350 con 0x7f4af8077c30 2024-06-18T02:31:02.171 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:01.560+0000 7f4adffff640 1 -- 172.21.15.1:0/4107709725 <== mgr.14160 v2:172.21.15.1:6800/85550937 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f4af80fe9d0 con 0x7f4ae0034730 2024-06-18T02:31:02.171 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:01.562+0000 7f4afe1e8640 1 -- 172.21.15.1:0/4107709725 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f4ae0034730 msgr2=0x7f4ae0036be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:31:02.171 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:01.562+0000 7f4afe1e8640 1 --2- 172.21.15.1:0/4107709725 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f4ae0034730 0x7f4ae0036be0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4aec006fd0 tx=0x7f4aec006db0 comp rx=0 tx=0).stop 2024-06-18T02:31:02.171 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:01.562+0000 7f4afe1e8640 1 -- 172.21.15.1:0/4107709725 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4af8077c30 msgr2=0x7f4af8112e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:31:02.171 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:01.562+0000 7f4afe1e8640 1 --2- 172.21.15.1:0/4107709725 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4af8077c30 0x7f4af8112e20 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4ae80317f0 tx=0x7f4ae802c040 comp rx=0 tx=0).stop 2024-06-18T02:31:02.172 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:01.562+0000 7f4afe1e8640 1 -- 172.21.15.1:0/4107709725 shutdown_connections 2024-06-18T02:31:02.172 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:01.562+0000 7f4afe1e8640 1 --2- 172.21.15.1:0/4107709725 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f4ae0034730 0x7f4ae0036be0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:31:02.172 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:01.562+0000 7f4afe1e8640 1 --2- 172.21.15.1:0/4107709725 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4af8077c30 0x7f4af8112e20 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:31:02.172 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:01.562+0000 7f4afe1e8640 1 -- 172.21.15.1:0/4107709725 >> 172.21.15.1:0/4107709725 conn(0x7f4af80fd5e0 msgr2=0x7f4af80fe200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:31:02.172 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:01.562+0000 7f4afe1e8640 1 -- 172.21.15.1:0/4107709725 shutdown_connections 2024-06-18T02:31:02.172 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:01.562+0000 7f4afe1e8640 1 -- 172.21.15.1:0/4107709725 wait complete. 2024-06-18T02:31:02.172 INFO:teuthology.orchestra.run.smithi001.stdout:Creating initial admin user... 2024-06-18T02:31:02.289 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:01 smithi001 ceph-mon[29099]: [18/Jun/2024:02:31:00] ENGINE Bus STARTING 2024-06-18T02:31:02.289 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:01 smithi001 ceph-mon[29099]: [18/Jun/2024:02:31:00] ENGINE Serving on https://172.21.15.1:7150 2024-06-18T02:31:02.289 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:01 smithi001 ceph-mon[29099]: [18/Jun/2024:02:31:00] ENGINE Bus STARTED 2024-06-18T02:31:02.289 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:01 smithi001 ceph-mon[29099]: [18/Jun/2024:02:31:00] ENGINE Error in HTTPServer.serve 2024-06-18T02:31:02.289 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Traceback (most recent call last): 2024-06-18T02:31:02.290 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-06-18T02:31:02.290 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: self._connections.run(self.expiration_interval) 2024-06-18T02:31:02.290 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-06-18T02:31:02.290 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: self._run(expiration_interval) 2024-06-18T02:31:02.290 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-06-18T02:31:02.290 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-06-18T02:31:02.290 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-06-18T02:31:02.290 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-06-18T02:31:02.290 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-06-18T02:31:02.290 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: s = self.context.wrap_socket( 2024-06-18T02:31:02.290 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-06-18T02:31:02.290 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: return self.sslsocket_class._create( 2024-06-18T02:31:02.290 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-06-18T02:31:02.291 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: self.do_handshake() 2024-06-18T02:31:02.291 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-06-18T02:31:02.291 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: self._sslobj.do_handshake() 2024-06-18T02:31:02.291 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1129) 2024-06-18T02:31:02.291 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:01 smithi001 ceph-mon[29099]: from='mgr.14160 172.21.15.1:0/1498115122' entity='mgr.smithi001.teoawz' 2024-06-18T02:31:02.291 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:01 smithi001 ceph-mon[29099]: mgrmap e12: smithi001.teoawz(active, since 2s) 2024-06-18T02:31:02.291 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:01 smithi001 ceph-mon[29099]: from='mgr.14160 172.21.15.1:0/1498115122' entity='mgr.smithi001.teoawz' 2024-06-18T02:31:02.291 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:01 smithi001 ceph-mon[29099]: from='mgr.14160 172.21.15.1:0/1498115122' entity='mgr.smithi001.teoawz' 2024-06-18T02:31:03.243 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:02 smithi001 ceph-mon[29099]: from='client.14172 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-06-18T02:31:03.796 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$Y2j/AtLv2NXktXMAU0O2OuA0zo4XnoTXc2ijrSuV/x4dpU.FGEWau", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1718677863, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-06-18T02:31:03.796 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:02.602+0000 7f7c407dd640 1 Processor -- start 2024-06-18T02:31:03.796 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:02.602+0000 7f7c407dd640 1 -- start start 2024-06-18T02:31:03.796 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:02.602+0000 7f7c407dd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c38075b70 0x7f7c381049d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:31:03.796 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:02.602+0000 7f7c407dd640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c38075f80 con 0x7f7c38075b70 2024-06-18T02:31:03.796 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:02.602+0000 7f7c3e579640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c38075b70 0x7f7c381049d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:31:03.796 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:02.602+0000 7f7c3e579640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c38075b70 0x7f7c381049d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42638/0 (socket says 172.21.15.1:42638) 2024-06-18T02:31:03.796 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:02.602+0000 7f7c3e579640 1 -- 172.21.15.1:0/2425014003 learned_addr learned my addr 172.21.15.1:0/2425014003 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:31:03.797 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:02.603+0000 7f7c3e579640 1 -- 172.21.15.1:0/2425014003 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c38104f10 con 0x7f7c38075b70 2024-06-18T02:31:03.797 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:02.604+0000 7f7c3e579640 1 --2- 172.21.15.1:0/2425014003 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c38075b70 0x7f7c381049d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f7c34009840 tx=0x7f7c340091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72024dd62280bf0c server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:31:03.797 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:02.604+0000 7f7c3d577640 1 -- 172.21.15.1:0/2425014003 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c34032440 con 0x7f7c38075b70 2024-06-18T02:31:03.797 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:02.604+0000 7f7c3d577640 1 -- 172.21.15.1:0/2425014003 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7c34032a70 con 0x7f7c38075b70 2024-06-18T02:31:03.797 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:02.604+0000 7f7c3d577640 1 -- 172.21.15.1:0/2425014003 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c34031ad0 con 0x7f7c38075b70 2024-06-18T02:31:03.797 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:02.605+0000 7f7c407dd640 1 -- 172.21.15.1:0/2425014003 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c38075b70 msgr2=0x7f7c381049d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:31:03.797 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:02.605+0000 7f7c407dd640 1 --2- 172.21.15.1:0/2425014003 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c38075b70 0x7f7c381049d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f7c34009840 tx=0x7f7c340091d0 comp rx=0 tx=0).stop 2024-06-18T02:31:03.797 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:02.605+0000 7f7c407dd640 1 -- 172.21.15.1:0/2425014003 shutdown_connections 2024-06-18T02:31:03.797 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:02.605+0000 7f7c407dd640 1 --2- 172.21.15.1:0/2425014003 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c38075b70 0x7f7c381049d0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:31:03.797 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:02.605+0000 7f7c407dd640 1 -- 172.21.15.1:0/2425014003 >> 172.21.15.1:0/2425014003 conn(0x7f7c380fc460 msgr2=0x7f7c380fe870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:31:03.797 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:02.605+0000 7f7c407dd640 1 -- 172.21.15.1:0/2425014003 shutdown_connections 2024-06-18T02:31:03.798 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:02.605+0000 7f7c407dd640 1 -- 172.21.15.1:0/2425014003 wait complete. 2024-06-18T02:31:03.798 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:02.605+0000 7f7c407dd640 1 Processor -- start 2024-06-18T02:31:03.798 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:02.605+0000 7f7c407dd640 1 -- start start 2024-06-18T02:31:03.798 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:02.606+0000 7f7c407dd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c38075b70 0x7f7c38110c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:31:03.798 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:02.606+0000 7f7c407dd640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c38111140 con 0x7f7c38075b70 2024-06-18T02:31:03.798 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:02.606+0000 7f7c3e579640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c38075b70 0x7f7c38110c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:31:03.798 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:02.606+0000 7f7c3e579640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c38075b70 0x7f7c38110c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42652/0 (socket says 172.21.15.1:42652) 2024-06-18T02:31:03.798 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:02.606+0000 7f7c3e579640 1 -- 172.21.15.1:0/2303449556 learned_addr learned my addr 172.21.15.1:0/2303449556 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:31:03.798 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:02.606+0000 7f7c3e579640 1 -- 172.21.15.1:0/2303449556 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c34009450 con 0x7f7c38075b70 2024-06-18T02:31:03.798 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:02.606+0000 7f7c3e579640 1 --2- 172.21.15.1:0/2303449556 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c38075b70 0x7f7c38110c00 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7c34031710 tx=0x7f7c3402c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:31:03.798 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:02.606+0000 7f7c2effd640 1 -- 172.21.15.1:0/2303449556 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c34031d40 con 0x7f7c38075b70 2024-06-18T02:31:03.798 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:02.607+0000 7f7c407dd640 1 -- 172.21.15.1:0/2303449556 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c38111340 con 0x7f7c38075b70 2024-06-18T02:31:03.799 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:02.607+0000 7f7c2effd640 1 -- 172.21.15.1:0/2303449556 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7c34031ea0 con 0x7f7c38075b70 2024-06-18T02:31:03.799 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:02.607+0000 7f7c2effd640 1 -- 172.21.15.1:0/2303449556 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c3403a910 con 0x7f7c38075b70 2024-06-18T02:31:03.799 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:02.607+0000 7f7c407dd640 1 -- 172.21.15.1:0/2303449556 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c381117b0 con 0x7f7c38075b70 2024-06-18T02:31:03.799 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:02.608+0000 7f7c407dd640 1 -- 172.21.15.1:0/2303449556 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c3804b480 con 0x7f7c38075b70 2024-06-18T02:31:03.799 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:02.608+0000 7f7c2effd640 1 -- 172.21.15.1:0/2303449556 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 12) v1 ==== 41959+0+0 (secure 0 0 0) 0x7f7c34039070 con 0x7f7c38075b70 2024-06-18T02:31:03.799 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:02.608+0000 7f7c2effd640 1 --2- 172.21.15.1:0/2303449556 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f7c240348b0 0x7f7c24036d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:31:03.799 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:02.608+0000 7f7c2effd640 1 -- 172.21.15.1:0/2303449556 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f7c34066e60 con 0x7f7c38075b70 2024-06-18T02:31:03.799 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:02.608+0000 7f7c3dd78640 1 --2- 172.21.15.1:0/2303449556 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f7c240348b0 0x7f7c24036d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:31:03.799 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:02.609+0000 7f7c3dd78640 1 --2- 172.21.15.1:0/2303449556 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f7c240348b0 0x7f7c24036d60 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7c28006fd0 tx=0x7f7c28006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:31:03.799 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:02.613+0000 7f7c2effd640 1 -- 172.21.15.1:0/2303449556 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f7c3404c350 con 0x7f7c38075b70 2024-06-18T02:31:03.799 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:02.804+0000 7f7c407dd640 1 -- 172.21.15.1:0/2303449556 --> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f7c3805e8c0 con 0x7f7c240348b0 2024-06-18T02:31:03.799 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:03.053+0000 7f7c2effd640 1 -- 172.21.15.1:0/2303449556 <== mgr.14160 v2:172.21.15.1:6800/85550937 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f7c3805e8c0 con 0x7f7c240348b0 2024-06-18T02:31:03.799 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:03.055+0000 7f7c407dd640 1 -- 172.21.15.1:0/2303449556 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f7c240348b0 msgr2=0x7f7c24036d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:31:03.800 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:03.055+0000 7f7c407dd640 1 --2- 172.21.15.1:0/2303449556 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f7c240348b0 0x7f7c24036d60 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7c28006fd0 tx=0x7f7c28006db0 comp rx=0 tx=0).stop 2024-06-18T02:31:03.800 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:03.055+0000 7f7c407dd640 1 -- 172.21.15.1:0/2303449556 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c38075b70 msgr2=0x7f7c38110c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:31:03.800 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:03.055+0000 7f7c407dd640 1 --2- 172.21.15.1:0/2303449556 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c38075b70 0x7f7c38110c00 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7c34031710 tx=0x7f7c3402c040 comp rx=0 tx=0).stop 2024-06-18T02:31:03.800 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:03.055+0000 7f7c407dd640 1 -- 172.21.15.1:0/2303449556 shutdown_connections 2024-06-18T02:31:03.800 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:03.055+0000 7f7c407dd640 1 --2- 172.21.15.1:0/2303449556 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f7c240348b0 0x7f7c24036d60 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:31:03.800 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:03.055+0000 7f7c407dd640 1 --2- 172.21.15.1:0/2303449556 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7c38075b70 0x7f7c38110c00 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:31:03.800 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:03.055+0000 7f7c407dd640 1 -- 172.21.15.1:0/2303449556 >> 172.21.15.1:0/2303449556 conn(0x7f7c380fc460 msgr2=0x7f7c380fd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:31:03.800 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:03.055+0000 7f7c407dd640 1 -- 172.21.15.1:0/2303449556 shutdown_connections 2024-06-18T02:31:03.800 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:03.055+0000 7f7c407dd640 1 -- 172.21.15.1:0/2303449556 wait complete. 2024-06-18T02:31:03.800 INFO:teuthology.orchestra.run.smithi001.stdout:Fetching dashboard port number... 2024-06-18T02:31:04.360 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:04 smithi001 ceph-mon[29099]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-06-18T02:31:04.360 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:04 smithi001 ceph-mon[29099]: from='mgr.14160 172.21.15.1:0/1498115122' entity='mgr.smithi001.teoawz' 2024-06-18T02:31:05.314 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:05 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/4271789095' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-06-18T02:31:05.327 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stdout 8443 2024-06-18T02:31:05.327 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:04.212+0000 7f6713205640 1 Processor -- start 2024-06-18T02:31:05.327 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:04.212+0000 7f6713205640 1 -- start start 2024-06-18T02:31:05.327 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:04.213+0000 7f6713205640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f670c075b00 0x7f670c104a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:31:05.327 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:04.213+0000 7f6713205640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f670c075f10 con 0x7f670c075b00 2024-06-18T02:31:05.328 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:04.213+0000 7f6710fa1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f670c075b00 0x7f670c104a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:31:05.328 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:04.213+0000 7f6710fa1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f670c075b00 0x7f670c104a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50232/0 (socket says 172.21.15.1:50232) 2024-06-18T02:31:05.328 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:04.213+0000 7f6710fa1640 1 -- 172.21.15.1:0/3888664113 learned_addr learned my addr 172.21.15.1:0/3888664113 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:31:05.328 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:04.213+0000 7f6710fa1640 1 -- 172.21.15.1:0/3888664113 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f670c104f40 con 0x7f670c075b00 2024-06-18T02:31:05.328 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:04.214+0000 7f6710fa1640 1 --2- 172.21.15.1:0/3888664113 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f670c075b00 0x7f670c104a00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6700009840 tx=0x7f67000091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e98d062cacfa920 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:31:05.329 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:04.214+0000 7f670b7fe640 1 -- 172.21.15.1:0/3888664113 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6700032440 con 0x7f670c075b00 2024-06-18T02:31:05.329 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:04.214+0000 7f670b7fe640 1 -- 172.21.15.1:0/3888664113 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6700032a70 con 0x7f670c075b00 2024-06-18T02:31:05.329 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:04.214+0000 7f670b7fe640 1 -- 172.21.15.1:0/3888664113 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6700031ad0 con 0x7f670c075b00 2024-06-18T02:31:05.329 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:04.214+0000 7f6713205640 1 -- 172.21.15.1:0/3888664113 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f670c075b00 msgr2=0x7f670c104a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:31:05.329 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:04.214+0000 7f6713205640 1 --2- 172.21.15.1:0/3888664113 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f670c075b00 0x7f670c104a00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6700009840 tx=0x7f67000091d0 comp rx=0 tx=0).stop 2024-06-18T02:31:05.329 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:04.215+0000 7f6713205640 1 -- 172.21.15.1:0/3888664113 shutdown_connections 2024-06-18T02:31:05.330 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:04.215+0000 7f6713205640 1 --2- 172.21.15.1:0/3888664113 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f670c075b00 0x7f670c104a00 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:31:05.330 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:04.215+0000 7f6713205640 1 -- 172.21.15.1:0/3888664113 >> 172.21.15.1:0/3888664113 conn(0x7f670c0fc450 msgr2=0x7f670c0fe880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:31:05.330 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:04.215+0000 7f6713205640 1 -- 172.21.15.1:0/3888664113 shutdown_connections 2024-06-18T02:31:05.330 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:04.215+0000 7f6713205640 1 -- 172.21.15.1:0/3888664113 wait complete. 2024-06-18T02:31:05.330 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:04.215+0000 7f6713205640 1 Processor -- start 2024-06-18T02:31:05.330 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:04.215+0000 7f6713205640 1 -- start start 2024-06-18T02:31:05.331 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:04.216+0000 7f6713205640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f670c075b00 0x7f670c110bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:31:05.331 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:04.216+0000 7f6713205640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f670c111100 con 0x7f670c075b00 2024-06-18T02:31:05.331 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:04.216+0000 7f6710fa1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f670c075b00 0x7f670c110bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:31:05.331 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:04.216+0000 7f6710fa1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f670c075b00 0x7f670c110bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50246/0 (socket says 172.21.15.1:50246) 2024-06-18T02:31:05.331 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:04.216+0000 7f6710fa1640 1 -- 172.21.15.1:0/4271789095 learned_addr learned my addr 172.21.15.1:0/4271789095 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:31:05.331 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:04.216+0000 7f6710fa1640 1 -- 172.21.15.1:0/4271789095 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6700009450 con 0x7f670c075b00 2024-06-18T02:31:05.332 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:04.216+0000 7f6710fa1640 1 --2- 172.21.15.1:0/4271789095 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f670c075b00 0x7f670c110bc0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f67000098a0 tx=0x7f670002c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:31:05.332 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:04.217+0000 7f67097fa640 1 -- 172.21.15.1:0/4271789095 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6700031e90 con 0x7f670c075b00 2024-06-18T02:31:05.332 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:04.217+0000 7f67097fa640 1 -- 172.21.15.1:0/4271789095 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f670003b910 con 0x7f670c075b00 2024-06-18T02:31:05.332 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:04.217+0000 7f67097fa640 1 -- 172.21.15.1:0/4271789095 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f670003a9b0 con 0x7f670c075b00 2024-06-18T02:31:05.332 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:04.217+0000 7f6713205640 1 -- 172.21.15.1:0/4271789095 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f670c111300 con 0x7f670c075b00 2024-06-18T02:31:05.332 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:04.218+0000 7f6713205640 1 -- 172.21.15.1:0/4271789095 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f670c111770 con 0x7f670c075b00 2024-06-18T02:31:05.333 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:04.219+0000 7f67097fa640 1 -- 172.21.15.1:0/4271789095 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 12) v1 ==== 41959+0+0 (secure 0 0 0) 0x7f6700039070 con 0x7f670c075b00 2024-06-18T02:31:05.333 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:04.219+0000 7f6713205640 1 -- 172.21.15.1:0/4271789095 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f670c04b480 con 0x7f670c075b00 2024-06-18T02:31:05.333 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:04.219+0000 7f67097fa640 1 --2- 172.21.15.1:0/4271789095 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f66fc034860 0x7f66fc036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:31:05.333 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:04.219+0000 7f67097fa640 1 -- 172.21.15.1:0/4271789095 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f6700066db0 con 0x7f670c075b00 2024-06-18T02:31:05.333 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:04.219+0000 7f670bfff640 1 --2- 172.21.15.1:0/4271789095 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f66fc034860 0x7f66fc036d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:31:05.334 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:04.220+0000 7f670bfff640 1 --2- 172.21.15.1:0/4271789095 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f66fc034860 0x7f66fc036d10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f66f8006fd0 tx=0x7f66f8006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:31:05.334 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:04.224+0000 7f67097fa640 1 -- 172.21.15.1:0/4271789095 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f6700039500 con 0x7f670c075b00 2024-06-18T02:31:05.334 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:04.402+0000 7f6713205640 1 -- 172.21.15.1:0/4271789095 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f670c1120d0 con 0x7f670c075b00 2024-06-18T02:31:05.334 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:04.402+0000 7f67097fa640 1 -- 172.21.15.1:0/4271789095 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f670003f070 con 0x7f670c075b00 2024-06-18T02:31:05.334 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:04.404+0000 7f6713205640 1 -- 172.21.15.1:0/4271789095 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f66fc034860 msgr2=0x7f66fc036d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:31:05.335 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:04.404+0000 7f6713205640 1 --2- 172.21.15.1:0/4271789095 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f66fc034860 0x7f66fc036d10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f66f8006fd0 tx=0x7f66f8006db0 comp rx=0 tx=0).stop 2024-06-18T02:31:05.335 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:04.404+0000 7f6713205640 1 -- 172.21.15.1:0/4271789095 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f670c075b00 msgr2=0x7f670c110bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:31:05.335 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:04.404+0000 7f6713205640 1 --2- 172.21.15.1:0/4271789095 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f670c075b00 0x7f670c110bc0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f67000098a0 tx=0x7f670002c040 comp rx=0 tx=0).stop 2024-06-18T02:31:05.335 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:04.404+0000 7f6713205640 1 -- 172.21.15.1:0/4271789095 shutdown_connections 2024-06-18T02:31:05.335 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:04.404+0000 7f6713205640 1 --2- 172.21.15.1:0/4271789095 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f66fc034860 0x7f66fc036d10 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:31:05.335 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:04.404+0000 7f6713205640 1 --2- 172.21.15.1:0/4271789095 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f670c075b00 0x7f670c110bc0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:31:05.336 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:04.404+0000 7f6713205640 1 -- 172.21.15.1:0/4271789095 >> 172.21.15.1:0/4271789095 conn(0x7f670c0fc450 msgr2=0x7f670c0fd0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:31:05.336 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:04.404+0000 7f6713205640 1 -- 172.21.15.1:0/4271789095 shutdown_connections 2024-06-18T02:31:05.336 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:04.404+0000 7f6713205640 1 -- 172.21.15.1:0/4271789095 wait complete. 2024-06-18T02:31:05.337 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-06-18T02:31:05.337 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stdout disabled 2024-06-18T02:31:05.342 INFO:teuthology.orchestra.run.smithi001.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-06-18T02:31:05.342 INFO:teuthology.orchestra.run.smithi001.stdout:systemctl: stdout inactive 2024-06-18T02:31:05.342 INFO:teuthology.orchestra.run.smithi001.stdout:firewalld.service is not enabled 2024-06-18T02:31:05.342 INFO:teuthology.orchestra.run.smithi001.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-06-18T02:31:05.344 INFO:teuthology.orchestra.run.smithi001.stdout:Ceph Dashboard is now available at: 2024-06-18T02:31:05.344 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:31:05.344 INFO:teuthology.orchestra.run.smithi001.stdout: URL: https://smithi001.front.sepia.ceph.com:8443/ 2024-06-18T02:31:05.344 INFO:teuthology.orchestra.run.smithi001.stdout: User: admin 2024-06-18T02:31:05.345 INFO:teuthology.orchestra.run.smithi001.stdout: Password: kztemrizp2 2024-06-18T02:31:05.345 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:31:05.345 INFO:teuthology.orchestra.run.smithi001.stdout:Saving cluster configuration to /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/config directory 2024-06-18T02:31:05.346 INFO:teuthology.orchestra.run.smithi001.stdout:Enabling autotune for osd_memory_target 2024-06-18T02:31:06.294 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:06 smithi001 ceph-mon[29099]: mgrmap e13: smithi001.teoawz(active, since 6s) 2024-06-18T02:31:06.578 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:05.752+0000 7fc8f3f13640 1 Processor -- start 2024-06-18T02:31:06.578 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:05.753+0000 7fc8f3f13640 1 -- start start 2024-06-18T02:31:06.578 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:05.753+0000 7fc8f3f13640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc8ec101270 0x7fc8ec101680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:31:06.578 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:05.753+0000 7fc8f3f13640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8ec101bc0 con 0x7fc8ec101270 2024-06-18T02:31:06.578 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:05.753+0000 7fc8f1caf640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc8ec101270 0x7fc8ec101680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:31:06.578 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:05.753+0000 7fc8f1caf640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc8ec101270 0x7fc8ec101680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50260/0 (socket says 172.21.15.1:50260) 2024-06-18T02:31:06.579 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:05.753+0000 7fc8f1caf640 1 -- 172.21.15.1:0/1005042198 learned_addr learned my addr 172.21.15.1:0/1005042198 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:31:06.579 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:05.753+0000 7fc8f1caf640 1 -- 172.21.15.1:0/1005042198 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8ec101d00 con 0x7fc8ec101270 2024-06-18T02:31:06.579 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:05.754+0000 7fc8f1caf640 1 --2- 172.21.15.1:0/1005042198 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc8ec101270 0x7fc8ec101680 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc8e8009840 tx=0x7fc8e80091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1ac52c64a2ee871 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:31:06.579 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:05.754+0000 7fc8f0cad640 1 -- 172.21.15.1:0/1005042198 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc8e8032440 con 0x7fc8ec101270 2024-06-18T02:31:06.579 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:05.754+0000 7fc8f0cad640 1 -- 172.21.15.1:0/1005042198 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc8e8032a70 con 0x7fc8ec101270 2024-06-18T02:31:06.579 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:05.754+0000 7fc8f0cad640 1 -- 172.21.15.1:0/1005042198 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc8e8031ad0 con 0x7fc8ec101270 2024-06-18T02:31:06.579 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:05.755+0000 7fc8f3f13640 1 -- 172.21.15.1:0/1005042198 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc8ec101270 msgr2=0x7fc8ec101680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:31:06.579 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:05.755+0000 7fc8f3f13640 1 --2- 172.21.15.1:0/1005042198 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc8ec101270 0x7fc8ec101680 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc8e8009840 tx=0x7fc8e80091d0 comp rx=0 tx=0).stop 2024-06-18T02:31:06.579 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:05.755+0000 7fc8f3f13640 1 -- 172.21.15.1:0/1005042198 shutdown_connections 2024-06-18T02:31:06.579 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:05.755+0000 7fc8f3f13640 1 --2- 172.21.15.1:0/1005042198 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc8ec101270 0x7fc8ec101680 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:31:06.579 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:05.755+0000 7fc8f3f13640 1 -- 172.21.15.1:0/1005042198 >> 172.21.15.1:0/1005042198 conn(0x7fc8ec0fbc20 msgr2=0x7fc8ec0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:31:06.579 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:05.755+0000 7fc8f3f13640 1 -- 172.21.15.1:0/1005042198 shutdown_connections 2024-06-18T02:31:06.579 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:05.755+0000 7fc8f3f13640 1 -- 172.21.15.1:0/1005042198 wait complete. 2024-06-18T02:31:06.580 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:05.756+0000 7fc8f3f13640 1 Processor -- start 2024-06-18T02:31:06.580 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:05.756+0000 7fc8f3f13640 1 -- start start 2024-06-18T02:31:06.580 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:05.756+0000 7fc8f3f13640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc8ec101270 0x7fc8ec106ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:31:06.580 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:05.756+0000 7fc8f3f13640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8ec107420 con 0x7fc8ec101270 2024-06-18T02:31:06.580 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:05.756+0000 7fc8f1caf640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc8ec101270 0x7fc8ec106ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:31:06.580 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:05.756+0000 7fc8f1caf640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc8ec101270 0x7fc8ec106ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50276/0 (socket says 172.21.15.1:50276) 2024-06-18T02:31:06.580 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:05.756+0000 7fc8f1caf640 1 -- 172.21.15.1:0/3977940866 learned_addr learned my addr 172.21.15.1:0/3977940866 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:31:06.580 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:05.756+0000 7fc8f1caf640 1 -- 172.21.15.1:0/3977940866 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8e8009450 con 0x7fc8ec101270 2024-06-18T02:31:06.580 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:05.756+0000 7fc8f1caf640 1 --2- 172.21.15.1:0/3977940866 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc8ec101270 0x7fc8ec106ee0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc8e8006a70 tx=0x7fc8e802c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:31:06.580 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:05.757+0000 7fc8e27fc640 1 -- 172.21.15.1:0/3977940866 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc8e8031e90 con 0x7fc8ec101270 2024-06-18T02:31:06.580 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:05.757+0000 7fc8e27fc640 1 -- 172.21.15.1:0/3977940866 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc8e803b910 con 0x7fc8ec101270 2024-06-18T02:31:06.580 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:05.757+0000 7fc8f3f13640 1 -- 172.21.15.1:0/3977940866 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8ec1054b0 con 0x7fc8ec101270 2024-06-18T02:31:06.580 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:05.757+0000 7fc8e27fc640 1 -- 172.21.15.1:0/3977940866 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc8e803a940 con 0x7fc8ec101270 2024-06-18T02:31:06.580 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:05.757+0000 7fc8f3f13640 1 -- 172.21.15.1:0/3977940866 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8ec105920 con 0x7fc8ec101270 2024-06-18T02:31:06.581 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:05.757+0000 7fc8e27fc640 1 -- 172.21.15.1:0/3977940866 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) v1 ==== 42001+0+0 (secure 0 0 0) 0x7fc8e8039070 con 0x7fc8ec101270 2024-06-18T02:31:06.581 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:05.758+0000 7fc8e27fc640 1 --2- 172.21.15.1:0/3977940866 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7fc8d8034950 0x7fc8d8036e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:31:06.581 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:05.758+0000 7fc8f3f13640 1 -- 172.21.15.1:0/3977940866 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8ec05ead0 con 0x7fc8ec101270 2024-06-18T02:31:06.581 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:05.758+0000 7fc8e27fc640 1 -- 172.21.15.1:0/3977940866 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fc8e8067040 con 0x7fc8ec101270 2024-06-18T02:31:06.581 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:05.758+0000 7fc8f14ae640 1 --2- 172.21.15.1:0/3977940866 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7fc8d8034950 0x7fc8d8036e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:31:06.581 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:05.758+0000 7fc8f14ae640 1 --2- 172.21.15.1:0/3977940866 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7fc8d8034950 0x7fc8d8036e00 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc006fd0 tx=0x7fc8dc006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:31:06.581 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:05.762+0000 7fc8e27fc640 1 -- 172.21.15.1:0/3977940866 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fc8e803f080 con 0x7fc8ec101270 2024-06-18T02:31:06.581 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:05.940+0000 7fc8f3f13640 1 -- 172.21.15.1:0/3977940866 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7fc8ec102b50 con 0x7fc8ec101270 2024-06-18T02:31:06.581 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:05.940+0000 7fc8e27fc640 1 -- 172.21.15.1:0/3977940866 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7fc8e8037070 con 0x7fc8ec101270 2024-06-18T02:31:06.581 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:05.942+0000 7fc8f3f13640 1 -- 172.21.15.1:0/3977940866 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7fc8d8034950 msgr2=0x7fc8d8036e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:31:06.581 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:05.942+0000 7fc8f3f13640 1 --2- 172.21.15.1:0/3977940866 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7fc8d8034950 0x7fc8d8036e00 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc006fd0 tx=0x7fc8dc006db0 comp rx=0 tx=0).stop 2024-06-18T02:31:06.581 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:05.942+0000 7fc8f3f13640 1 -- 172.21.15.1:0/3977940866 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc8ec101270 msgr2=0x7fc8ec106ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:31:06.581 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:05.942+0000 7fc8f3f13640 1 --2- 172.21.15.1:0/3977940866 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc8ec101270 0x7fc8ec106ee0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc8e8006a70 tx=0x7fc8e802c040 comp rx=0 tx=0).stop 2024-06-18T02:31:06.582 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:05.942+0000 7fc8f3f13640 1 -- 172.21.15.1:0/3977940866 shutdown_connections 2024-06-18T02:31:06.582 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:05.942+0000 7fc8f3f13640 1 --2- 172.21.15.1:0/3977940866 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7fc8d8034950 0x7fc8d8036e00 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:31:06.582 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:05.942+0000 7fc8f3f13640 1 --2- 172.21.15.1:0/3977940866 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc8ec101270 0x7fc8ec106ee0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:31:06.582 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:05.942+0000 7fc8f3f13640 1 -- 172.21.15.1:0/3977940866 >> 172.21.15.1:0/3977940866 conn(0x7fc8ec0fbc20 msgr2=0x7fc8ec0fcb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:31:06.582 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:05.942+0000 7fc8f3f13640 1 -- 172.21.15.1:0/3977940866 shutdown_connections 2024-06-18T02:31:06.582 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:05.942+0000 7fc8f3f13640 1 -- 172.21.15.1:0/3977940866 wait complete. 2024-06-18T02:31:07.876 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:06.962+0000 7f81af5b5640 1 Processor -- start 2024-06-18T02:31:07.876 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:06.962+0000 7f81af5b5640 1 -- start start 2024-06-18T02:31:07.877 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:06.962+0000 7f81af5b5640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81a8101d00 0x7f81a8102110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:31:07.877 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:06.962+0000 7f81af5b5640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81a8102650 con 0x7f81a8101d00 2024-06-18T02:31:07.877 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:06.962+0000 7f81ad351640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81a8101d00 0x7f81a8102110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:31:07.877 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:06.962+0000 7f81ad351640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81a8101d00 0x7f81a8102110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50290/0 (socket says 172.21.15.1:50290) 2024-06-18T02:31:07.877 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:06.962+0000 7f81ad351640 1 -- 172.21.15.1:0/3320272830 learned_addr learned my addr 172.21.15.1:0/3320272830 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:31:07.877 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:06.963+0000 7f81ad351640 1 -- 172.21.15.1:0/3320272830 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81a8102790 con 0x7f81a8101d00 2024-06-18T02:31:07.877 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:06.963+0000 7f81ad351640 1 --2- 172.21.15.1:0/3320272830 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81a8101d00 0x7f81a8102110 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f81a4009840 tx=0x7f81a40091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=728d0f5ecba5cc35 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:31:07.877 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:06.963+0000 7f819ffff640 1 -- 172.21.15.1:0/3320272830 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f81a4032440 con 0x7f81a8101d00 2024-06-18T02:31:07.877 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:06.963+0000 7f819ffff640 1 -- 172.21.15.1:0/3320272830 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f81a4032a70 con 0x7f81a8101d00 2024-06-18T02:31:07.877 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:06.963+0000 7f819ffff640 1 -- 172.21.15.1:0/3320272830 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f81a4031ad0 con 0x7f81a8101d00 2024-06-18T02:31:07.877 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:06.964+0000 7f81af5b5640 1 -- 172.21.15.1:0/3320272830 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81a8101d00 msgr2=0x7f81a8102110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:31:07.877 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:06.964+0000 7f81af5b5640 1 --2- 172.21.15.1:0/3320272830 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81a8101d00 0x7f81a8102110 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f81a4009840 tx=0x7f81a40091d0 comp rx=0 tx=0).stop 2024-06-18T02:31:07.877 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:06.964+0000 7f81af5b5640 1 -- 172.21.15.1:0/3320272830 shutdown_connections 2024-06-18T02:31:07.878 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:06.964+0000 7f81af5b5640 1 --2- 172.21.15.1:0/3320272830 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81a8101d00 0x7f81a8102110 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:31:07.878 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:06.964+0000 7f81af5b5640 1 -- 172.21.15.1:0/3320272830 >> 172.21.15.1:0/3320272830 conn(0x7f81a80fd500 msgr2=0x7f81a80ff910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:31:07.878 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:06.964+0000 7f81af5b5640 1 -- 172.21.15.1:0/3320272830 shutdown_connections 2024-06-18T02:31:07.878 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:06.964+0000 7f81af5b5640 1 -- 172.21.15.1:0/3320272830 wait complete. 2024-06-18T02:31:07.878 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:06.965+0000 7f81af5b5640 1 Processor -- start 2024-06-18T02:31:07.878 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:06.965+0000 7f81af5b5640 1 -- start start 2024-06-18T02:31:07.878 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:06.965+0000 7f81af5b5640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81a8101d00 0x7f81a8077c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:31:07.878 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:06.965+0000 7f81af5b5640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81a8078180 con 0x7f81a8101d00 2024-06-18T02:31:07.878 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:06.965+0000 7f81ad351640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81a8101d00 0x7f81a8077c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:31:07.878 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:06.965+0000 7f81ad351640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81a8101d00 0x7f81a8077c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50292/0 (socket says 172.21.15.1:50292) 2024-06-18T02:31:07.878 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:06.965+0000 7f81ad351640 1 -- 172.21.15.1:0/377755786 learned_addr learned my addr 172.21.15.1:0/377755786 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:31:07.878 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:06.965+0000 7f81ad351640 1 -- 172.21.15.1:0/377755786 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81a4009450 con 0x7f81a8101d00 2024-06-18T02:31:07.879 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:06.965+0000 7f81ad351640 1 --2- 172.21.15.1:0/377755786 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81a8101d00 0x7f81a8077c40 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f81a4006a70 tx=0x7f81a402c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:31:07.879 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:06.965+0000 7f819dffb640 1 -- 172.21.15.1:0/377755786 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f81a4031e90 con 0x7f81a8101d00 2024-06-18T02:31:07.879 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:06.965+0000 7f819dffb640 1 -- 172.21.15.1:0/377755786 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f81a403b910 con 0x7f81a8101d00 2024-06-18T02:31:07.879 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:06.965+0000 7f81af5b5640 1 -- 172.21.15.1:0/377755786 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81a8076210 con 0x7f81a8101d00 2024-06-18T02:31:07.879 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:06.965+0000 7f819dffb640 1 -- 172.21.15.1:0/377755786 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f81a403a940 con 0x7f81a8101d00 2024-06-18T02:31:07.879 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:06.966+0000 7f81af5b5640 1 -- 172.21.15.1:0/377755786 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81a8076680 con 0x7f81a8101d00 2024-06-18T02:31:07.879 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:06.966+0000 7f819dffb640 1 -- 172.21.15.1:0/377755786 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) v1 ==== 42001+0+0 (secure 0 0 0) 0x7f81a4039070 con 0x7f81a8101d00 2024-06-18T02:31:07.879 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:06.967+0000 7f819dffb640 1 --2- 172.21.15.1:0/377755786 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f8194034900 0x7f8194036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:31:07.879 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:06.967+0000 7f819dffb640 1 -- 172.21.15.1:0/377755786 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f81a4067060 con 0x7f81a8101d00 2024-06-18T02:31:07.879 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:06.967+0000 7f81af5b5640 1 -- 172.21.15.1:0/377755786 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81a805ead0 con 0x7f81a8101d00 2024-06-18T02:31:07.879 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:06.967+0000 7f81acb50640 1 --2- 172.21.15.1:0/377755786 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f8194034900 0x7f8194036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:31:07.879 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:06.967+0000 7f81acb50640 1 --2- 172.21.15.1:0/377755786 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f8194034900 0x7f8194036db0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f8198006fd0 tx=0x7f8198006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:31:07.879 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:06.971+0000 7f819dffb640 1 -- 172.21.15.1:0/377755786 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f81a403f080 con 0x7f81a8101d00 2024-06-18T02:31:07.880 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:07.234+0000 7f81af5b5640 1 -- 172.21.15.1:0/377755786 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f81a8103d00 con 0x7f81a8101d00 2024-06-18T02:31:07.880 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:07.236+0000 7f819dffb640 1 -- 172.21.15.1:0/377755786 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v33)=0 set mgr/dashboard/cluster/status v33) v1 ==== 153+0+0 (secure 0 0 0) 0x7f81a4037070 con 0x7f81a8101d00 2024-06-18T02:31:07.880 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-06-18T02:31:07.880 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:07.238+0000 7f81af5b5640 1 -- 172.21.15.1:0/377755786 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f8194034900 msgr2=0x7f8194036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:31:07.880 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:07.238+0000 7f81af5b5640 1 --2- 172.21.15.1:0/377755786 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f8194034900 0x7f8194036db0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f8198006fd0 tx=0x7f8198006db0 comp rx=0 tx=0).stop 2024-06-18T02:31:07.880 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:07.238+0000 7f81af5b5640 1 -- 172.21.15.1:0/377755786 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81a8101d00 msgr2=0x7f81a8077c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:31:07.880 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:07.238+0000 7f81af5b5640 1 --2- 172.21.15.1:0/377755786 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81a8101d00 0x7f81a8077c40 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f81a4006a70 tx=0x7f81a402c040 comp rx=0 tx=0).stop 2024-06-18T02:31:07.880 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:07.238+0000 7f81af5b5640 1 -- 172.21.15.1:0/377755786 shutdown_connections 2024-06-18T02:31:07.880 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:07.238+0000 7f81af5b5640 1 --2- 172.21.15.1:0/377755786 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f8194034900 0x7f8194036db0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:31:07.880 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:07.238+0000 7f81af5b5640 1 --2- 172.21.15.1:0/377755786 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81a8101d00 0x7f81a8077c40 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:31:07.880 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:07.238+0000 7f81af5b5640 1 -- 172.21.15.1:0/377755786 >> 172.21.15.1:0/377755786 conn(0x7f81a80fd500 msgr2=0x7f81a80fe290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:31:07.880 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:07.239+0000 7f81af5b5640 1 -- 172.21.15.1:0/377755786 shutdown_connections 2024-06-18T02:31:07.880 INFO:teuthology.orchestra.run.smithi001.stdout:/usr/bin/ceph: stderr 2024-06-18T02:31:07.239+0000 7f81af5b5640 1 -- 172.21.15.1:0/377755786 wait complete. 2024-06-18T02:31:07.881 INFO:teuthology.orchestra.run.smithi001.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-06-18T02:31:07.881 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:31:07.881 INFO:teuthology.orchestra.run.smithi001.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-06-18T02:31:07.881 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:31:07.881 INFO:teuthology.orchestra.run.smithi001.stdout:Or, if you are only running a single cluster on this host: 2024-06-18T02:31:07.881 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:31:07.881 INFO:teuthology.orchestra.run.smithi001.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-06-18T02:31:07.881 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:31:07.881 INFO:teuthology.orchestra.run.smithi001.stdout:Please consider enabling telemetry to help improve Ceph: 2024-06-18T02:31:07.881 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:31:07.881 INFO:teuthology.orchestra.run.smithi001.stdout: ceph telemetry on 2024-06-18T02:31:07.881 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:31:07.881 INFO:teuthology.orchestra.run.smithi001.stdout:For more information see: 2024-06-18T02:31:07.882 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:31:07.882 INFO:teuthology.orchestra.run.smithi001.stdout: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-06-18T02:31:07.882 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:31:07.882 INFO:teuthology.orchestra.run.smithi001.stdout:Bootstrap complete. 2024-06-18T02:31:07.912 INFO:tasks.cephadm:Fetching config... 2024-06-18T02:31:07.912 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-06-18T02:31:07.913 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-06-18T02:31:07.929 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-06-18T02:31:07.930 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-06-18T02:31:07.930 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-06-18T02:31:07.986 INFO:tasks.cephadm:Fetching mon keyring... 2024-06-18T02:31:07.986 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-06-18T02:31:07.987 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/keyring of=/dev/stdout 2024-06-18T02:31:08.053 INFO:tasks.cephadm:Fetching pub ssh key... 2024-06-18T02:31:08.053 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-06-18T02:31:08.053 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-06-18T02:31:08.109 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-06-18T02:31:08.109 DEBUG:teuthology.orchestra.run.smithi001:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDVBZQGul2r4JowODwEUyOlWTzn0CqoGU0SOPOVPnFp/5+EFQlYXFGXpzHx6xz3BIkFoo/k86YCxPAT5+6kJwj98TvvCtchAgLk2IZ9oVdnXURVwDaL/4RVUyGle/8Z/mJ6uHvyO2eyK/jJPYjWhBPCUVx1ah0BoAi46VdBDLmnOSijEHLX3Fc1rMRMvMOjNWuyfszOwVYiqXARHBQk50mZIl+N+/8levnS+9Riy3zeGmqbHutBwtTFn7N4zu+DyV95pcUk+8kWp1uf9JJCnVD4kl1DmKpe11Cvhv3hUvTiYs2QN/kZRHzn2wjI3ODkd1gx/i2uDqnCZj61bf24XRg6QUuNMx4Gt5aNe7G/qQTIsuTGnGzRMkd5aUYS74AUOPhtMisYilCQ1r7LoQ3e+opANXy91o0j0zq6hrisKoTWom6CNcSVIGDEupRImE9jPRlYtb8rCr/T7jBtFac10sLtR4G23/ShCBPCZqk3KhOJbp5xboiT3h5wbk0T4RkrL+8= ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-06-18T02:31:08.184 INFO:teuthology.orchestra.run.smithi001.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDVBZQGul2r4JowODwEUyOlWTzn0CqoGU0SOPOVPnFp/5+EFQlYXFGXpzHx6xz3BIkFoo/k86YCxPAT5+6kJwj98TvvCtchAgLk2IZ9oVdnXURVwDaL/4RVUyGle/8Z/mJ6uHvyO2eyK/jJPYjWhBPCUVx1ah0BoAi46VdBDLmnOSijEHLX3Fc1rMRMvMOjNWuyfszOwVYiqXARHBQk50mZIl+N+/8levnS+9Riy3zeGmqbHutBwtTFn7N4zu+DyV95pcUk+8kWp1uf9JJCnVD4kl1DmKpe11Cvhv3hUvTiYs2QN/kZRHzn2wjI3ODkd1gx/i2uDqnCZj61bf24XRg6QUuNMx4Gt5aNe7G/qQTIsuTGnGzRMkd5aUYS74AUOPhtMisYilCQ1r7LoQ3e+opANXy91o0j0zq6hrisKoTWom6CNcSVIGDEupRImE9jPRlYtb8rCr/T7jBtFac10sLtR4G23/ShCBPCZqk3KhOJbp5xboiT3h5wbk0T4RkrL+8= ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968 2024-06-18T02:31:08.196 DEBUG:teuthology.orchestra.run.smithi028:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDVBZQGul2r4JowODwEUyOlWTzn0CqoGU0SOPOVPnFp/5+EFQlYXFGXpzHx6xz3BIkFoo/k86YCxPAT5+6kJwj98TvvCtchAgLk2IZ9oVdnXURVwDaL/4RVUyGle/8Z/mJ6uHvyO2eyK/jJPYjWhBPCUVx1ah0BoAi46VdBDLmnOSijEHLX3Fc1rMRMvMOjNWuyfszOwVYiqXARHBQk50mZIl+N+/8levnS+9Riy3zeGmqbHutBwtTFn7N4zu+DyV95pcUk+8kWp1uf9JJCnVD4kl1DmKpe11Cvhv3hUvTiYs2QN/kZRHzn2wjI3ODkd1gx/i2uDqnCZj61bf24XRg6QUuNMx4Gt5aNe7G/qQTIsuTGnGzRMkd5aUYS74AUOPhtMisYilCQ1r7LoQ3e+opANXy91o0j0zq6hrisKoTWom6CNcSVIGDEupRImE9jPRlYtb8rCr/T7jBtFac10sLtR4G23/ShCBPCZqk3KhOJbp5xboiT3h5wbk0T4RkrL+8= ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-06-18T02:31:08.241 INFO:teuthology.orchestra.run.smithi028.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDVBZQGul2r4JowODwEUyOlWTzn0CqoGU0SOPOVPnFp/5+EFQlYXFGXpzHx6xz3BIkFoo/k86YCxPAT5+6kJwj98TvvCtchAgLk2IZ9oVdnXURVwDaL/4RVUyGle/8Z/mJ6uHvyO2eyK/jJPYjWhBPCUVx1ah0BoAi46VdBDLmnOSijEHLX3Fc1rMRMvMOjNWuyfszOwVYiqXARHBQk50mZIl+N+/8levnS+9Riy3zeGmqbHutBwtTFn7N4zu+DyV95pcUk+8kWp1uf9JJCnVD4kl1DmKpe11Cvhv3hUvTiYs2QN/kZRHzn2wjI3ODkd1gx/i2uDqnCZj61bf24XRg6QUuNMx4Gt5aNe7G/qQTIsuTGnGzRMkd5aUYS74AUOPhtMisYilCQ1r7LoQ3e+opANXy91o0j0zq6hrisKoTWom6CNcSVIGDEupRImE9jPRlYtb8rCr/T7jBtFac10sLtR4G23/ShCBPCZqk3KhOJbp5xboiT3h5wbk0T4RkrL+8= ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968 2024-06-18T02:31:08.254 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-06-18T02:31:08.439 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:08 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/377755786' entity='client.admin' 2024-06-18T02:31:08.522 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:31:10.990 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:10 smithi001 ceph-mon[29099]: from='mgr.14160 172.21.15.1:0/1498115122' entity='mgr.smithi001.teoawz' 2024-06-18T02:31:10.990 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:10 smithi001 ceph-mon[29099]: from='mgr.14160 172.21.15.1:0/1498115122' entity='mgr.smithi001.teoawz' 2024-06-18T02:31:10.990 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:10 smithi001 ceph-mon[29099]: from='mgr.14160 172.21.15.1:0/1498115122' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config rm", "who": "osd/host:smithi001", "name": "osd_memory_target"}]: dispatch 2024-06-18T02:31:10.990 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:10 smithi001 ceph-mon[29099]: from='mgr.14160 172.21.15.1:0/1498115122' entity='mgr.smithi001.teoawz' 2024-06-18T02:31:10.990 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:10 smithi001 ceph-mon[29099]: from='mgr.14160 172.21.15.1:0/1498115122' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi001", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-06-18T02:31:10.990 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:10 smithi001 ceph-mon[29099]: from='mgr.14160 172.21.15.1:0/1498115122' entity='mgr.smithi001.teoawz' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi001", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-06-18T02:31:10.990 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:10 smithi001 ceph-mon[29099]: from='mgr.14160 172.21.15.1:0/1498115122' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:31:10.990 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:10 smithi001 ceph-mon[29099]: Deploying daemon crash.smithi001 on smithi001 2024-06-18T02:31:11.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:11.736+0000 7fd3ba4f7640 1 -- 172.21.15.1:0/2275608574 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3b40faed0 msgr2=0x7fd3b40fb2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:31:11.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:11.736+0000 7fd3ba4f7640 1 --2- 172.21.15.1:0/2275608574 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3b40faed0 0x7fd3b40fb2e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd3a4004850 tx=0x7fd3a4009da0 comp rx=0 tx=0).stop 2024-06-18T02:31:11.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:11.736+0000 7fd3ba4f7640 1 -- 172.21.15.1:0/2275608574 shutdown_connections 2024-06-18T02:31:11.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:11.736+0000 7fd3ba4f7640 1 --2- 172.21.15.1:0/2275608574 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3b40faed0 0x7fd3b40fb2e0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:31:11.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:11.736+0000 7fd3ba4f7640 1 -- 172.21.15.1:0/2275608574 >> 172.21.15.1:0/2275608574 conn(0x7fd3b40f65f0 msgr2=0x7fd3b40f8a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:31:11.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:11.737+0000 7fd3ba4f7640 1 -- 172.21.15.1:0/2275608574 shutdown_connections 2024-06-18T02:31:11.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:11.737+0000 7fd3ba4f7640 1 -- 172.21.15.1:0/2275608574 wait complete. 2024-06-18T02:31:11.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:11.737+0000 7fd3ba4f7640 1 Processor -- start 2024-06-18T02:31:11.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:11.737+0000 7fd3ba4f7640 1 -- start start 2024-06-18T02:31:11.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:11.737+0000 7fd3ba4f7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3b40faed0 0x7fd3b410e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:31:11.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:11.737+0000 7fd3ba4f7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b410ec70 con 0x7fd3b40faed0 2024-06-18T02:31:11.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:11.737+0000 7fd3b3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3b40faed0 0x7fd3b410e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:31:11.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:11.737+0000 7fd3b3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3b40faed0 0x7fd3b410e730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50310/0 (socket says 172.21.15.1:50310) 2024-06-18T02:31:11.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:11.737+0000 7fd3b3fff640 1 -- 172.21.15.1:0/1031294988 learned_addr learned my addr 172.21.15.1:0/1031294988 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:31:11.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:11.738+0000 7fd3b3fff640 1 -- 172.21.15.1:0/1031294988 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3a4005040 con 0x7fd3b40faed0 2024-06-18T02:31:11.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:11.738+0000 7fd3b3fff640 1 --2- 172.21.15.1:0/1031294988 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3b40faed0 0x7fd3b410e730 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd3a4004820 tx=0x7fd3a4009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:31:11.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:11.738+0000 7fd3b17fa640 1 -- 172.21.15.1:0/1031294988 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd3a4032650 con 0x7fd3b40faed0 2024-06-18T02:31:11.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:11.738+0000 7fd3b17fa640 1 -- 172.21.15.1:0/1031294988 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd3a4032c80 con 0x7fd3b40faed0 2024-06-18T02:31:11.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:11.738+0000 7fd3ba4f7640 1 -- 172.21.15.1:0/1031294988 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3b410ee70 con 0x7fd3b40faed0 2024-06-18T02:31:11.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:11.738+0000 7fd3b17fa640 1 -- 172.21.15.1:0/1031294988 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd3a403eda0 con 0x7fd3b40faed0 2024-06-18T02:31:11.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:11.738+0000 7fd3ba4f7640 1 -- 172.21.15.1:0/1031294988 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3b410fa00 con 0x7fd3b40faed0 2024-06-18T02:31:11.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:11.739+0000 7fd3b17fa640 1 -- 172.21.15.1:0/1031294988 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) v1 ==== 42001+0+0 (secure 0 0 0) 0x7fd3a4030020 con 0x7fd3b40faed0 2024-06-18T02:31:11.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:11.739+0000 7fd3b17fa640 1 --2- 172.21.15.1:0/1031294988 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7fd3a0034900 0x7fd3a0036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:31:11.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:11.739+0000 7fd3b17fa640 1 -- 172.21.15.1:0/1031294988 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fd3a4066920 con 0x7fd3b40faed0 2024-06-18T02:31:11.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:11.739+0000 7fd3ba4f7640 1 -- 172.21.15.1:0/1031294988 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3b410f340 con 0x7fd3b40faed0 2024-06-18T02:31:11.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:11.739+0000 7fd3abfff640 1 --2- 172.21.15.1:0/1031294988 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7fd3a0034900 0x7fd3a0036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:31:11.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:11.740+0000 7fd3abfff640 1 --2- 172.21.15.1:0/1031294988 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7fd3a0034900 0x7fd3a0036db0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd39c006fd0 tx=0x7fd39c006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:31:11.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:11.744+0000 7fd3b17fa640 1 -- 172.21.15.1:0/1031294988 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fd3b410f340 con 0x7fd3b40faed0 2024-06-18T02:31:11.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:11.922+0000 7fd3ba4f7640 1 -- 172.21.15.1:0/1031294988 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7fd3b410f340 con 0x7fd3b40faed0 2024-06-18T02:31:11.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:11.925+0000 7fd3b17fa640 1 -- 172.21.15.1:0/1031294988 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7fd3b410f340 con 0x7fd3b40faed0 2024-06-18T02:31:11.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:11.927+0000 7fd3ba4f7640 1 -- 172.21.15.1:0/1031294988 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7fd3a0034900 msgr2=0x7fd3a0036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:31:11.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:11.927+0000 7fd3ba4f7640 1 --2- 172.21.15.1:0/1031294988 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7fd3a0034900 0x7fd3a0036db0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd39c006fd0 tx=0x7fd39c006db0 comp rx=0 tx=0).stop 2024-06-18T02:31:11.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:11.927+0000 7fd3ba4f7640 1 -- 172.21.15.1:0/1031294988 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3b40faed0 msgr2=0x7fd3b410e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:31:11.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:11.927+0000 7fd3ba4f7640 1 --2- 172.21.15.1:0/1031294988 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3b40faed0 0x7fd3b410e730 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd3a4004820 tx=0x7fd3a4009da0 comp rx=0 tx=0).stop 2024-06-18T02:31:11.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:11.927+0000 7fd3ba4f7640 1 -- 172.21.15.1:0/1031294988 shutdown_connections 2024-06-18T02:31:11.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:11.928+0000 7fd3ba4f7640 1 --2- 172.21.15.1:0/1031294988 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7fd3a0034900 0x7fd3a0036db0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:31:11.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:11.928+0000 7fd3ba4f7640 1 --2- 172.21.15.1:0/1031294988 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3b40faed0 0x7fd3b410e730 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:31:11.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:11.928+0000 7fd3ba4f7640 1 -- 172.21.15.1:0/1031294988 >> 172.21.15.1:0/1031294988 conn(0x7fd3b40f65f0 msgr2=0x7fd3b40f8a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:31:11.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:11.928+0000 7fd3ba4f7640 1 -- 172.21.15.1:0/1031294988 shutdown_connections 2024-06-18T02:31:11.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:11.928+0000 7fd3ba4f7640 1 -- 172.21.15.1:0/1031294988 wait complete. 2024-06-18T02:31:12.973 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-06-18T02:31:12.973 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-06-18T02:31:13.254 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:12 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/1031294988' entity='client.admin' 2024-06-18T02:31:13.480 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:31:15.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:15.069+0000 7f4b1d235640 1 -- 172.21.15.1:0/3061041338 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b180fa370 msgr2=0x7f4b180fa780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:31:15.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:15.069+0000 7f4b1d235640 1 --2- 172.21.15.1:0/3061041338 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b180fa370 0x7f4b180fa780 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4b10004850 tx=0x7f4b10009da0 comp rx=0 tx=0).stop 2024-06-18T02:31:15.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:15.069+0000 7f4b1d235640 1 -- 172.21.15.1:0/3061041338 shutdown_connections 2024-06-18T02:31:15.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:15.069+0000 7f4b1d235640 1 --2- 172.21.15.1:0/3061041338 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b180fa370 0x7f4b180fa780 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:31:15.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:15.069+0000 7f4b1d235640 1 -- 172.21.15.1:0/3061041338 >> 172.21.15.1:0/3061041338 conn(0x7f4b180f5d70 msgr2=0x7f4b180f81a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:31:15.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:15.069+0000 7f4b1d235640 1 -- 172.21.15.1:0/3061041338 shutdown_connections 2024-06-18T02:31:15.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:15.069+0000 7f4b1d235640 1 -- 172.21.15.1:0/3061041338 wait complete. 2024-06-18T02:31:15.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:15.070+0000 7f4b1d235640 1 Processor -- start 2024-06-18T02:31:15.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:15.070+0000 7f4b1d235640 1 -- start start 2024-06-18T02:31:15.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:15.070+0000 7f4b1d235640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b180fa370 0x7f4b180fcda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:31:15.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:15.070+0000 7f4b1d235640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b180fd2e0 con 0x7f4b180fa370 2024-06-18T02:31:15.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:15.070+0000 7f4b16d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b180fa370 0x7f4b180fcda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:31:15.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:15.071+0000 7f4b16d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b180fa370 0x7f4b180fcda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:32822/0 (socket says 172.21.15.1:32822) 2024-06-18T02:31:15.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:15.071+0000 7f4b16d9d640 1 -- 172.21.15.1:0/4119246600 learned_addr learned my addr 172.21.15.1:0/4119246600 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:31:15.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:15.071+0000 7f4b16d9d640 1 -- 172.21.15.1:0/4119246600 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b10005040 con 0x7f4b180fa370 2024-06-18T02:31:15.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:15.071+0000 7f4b16d9d640 1 --2- 172.21.15.1:0/4119246600 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b180fa370 0x7f4b180fcda0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f4b10000c00 tx=0x7f4b10009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:31:15.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:15.071+0000 7f4b0f7fe640 1 -- 172.21.15.1:0/4119246600 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b10032650 con 0x7f4b180fa370 2024-06-18T02:31:15.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:15.071+0000 7f4b1d235640 1 -- 172.21.15.1:0/4119246600 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b180fb370 con 0x7f4b180fa370 2024-06-18T02:31:15.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:15.071+0000 7f4b1d235640 1 -- 172.21.15.1:0/4119246600 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b180fb840 con 0x7f4b180fa370 2024-06-18T02:31:15.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:15.071+0000 7f4b0f7fe640 1 -- 172.21.15.1:0/4119246600 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4b10032c80 con 0x7f4b180fa370 2024-06-18T02:31:15.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:15.071+0000 7f4b0f7fe640 1 -- 172.21.15.1:0/4119246600 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b1003dcb0 con 0x7f4b180fa370 2024-06-18T02:31:15.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:15.072+0000 7f4b0f7fe640 1 -- 172.21.15.1:0/4119246600 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) v1 ==== 42001+0+0 (secure 0 0 0) 0x7f4b10030020 con 0x7f4b180fa370 2024-06-18T02:31:15.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:15.072+0000 7f4b1d235640 1 -- 172.21.15.1:0/4119246600 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b1805ead0 con 0x7f4b180fa370 2024-06-18T02:31:15.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:15.072+0000 7f4b0f7fe640 1 --2- 172.21.15.1:0/4119246600 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f4b04034900 0x7f4b04036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:31:15.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:15.072+0000 7f4b0f7fe640 1 -- 172.21.15.1:0/4119246600 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f4b10066650 con 0x7f4b180fa370 2024-06-18T02:31:15.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:15.072+0000 7f4b1659c640 1 --2- 172.21.15.1:0/4119246600 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f4b04034900 0x7f4b04036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:31:15.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:15.073+0000 7f4b1659c640 1 --2- 172.21.15.1:0/4119246600 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f4b04034900 0x7f4b04036db0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4b00006fd0 tx=0x7f4b00006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:31:15.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:15.077+0000 7f4b0f7fe640 1 -- 172.21.15.1:0/4119246600 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f4b10033420 con 0x7f4b180fa370 2024-06-18T02:31:15.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:15.261+0000 7f4b1d235640 1 -- 172.21.15.1:0/4119246600 --> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f4b181059f0 con 0x7f4b04034900 2024-06-18T02:31:15.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:15.264+0000 7f4b0f7fe640 1 -- 172.21.15.1:0/4119246600 <== mgr.14160 v2:172.21.15.1:6800/85550937 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f4b181059f0 con 0x7f4b04034900 2024-06-18T02:31:15.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:15.266+0000 7f4b1d235640 1 -- 172.21.15.1:0/4119246600 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f4b04034900 msgr2=0x7f4b04036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:31:15.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:15.266+0000 7f4b1d235640 1 --2- 172.21.15.1:0/4119246600 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f4b04034900 0x7f4b04036db0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4b00006fd0 tx=0x7f4b00006db0 comp rx=0 tx=0).stop 2024-06-18T02:31:15.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:15.266+0000 7f4b1d235640 1 -- 172.21.15.1:0/4119246600 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b180fa370 msgr2=0x7f4b180fcda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:31:15.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:15.266+0000 7f4b1d235640 1 --2- 172.21.15.1:0/4119246600 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b180fa370 0x7f4b180fcda0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f4b10000c00 tx=0x7f4b10009da0 comp rx=0 tx=0).stop 2024-06-18T02:31:15.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:15.266+0000 7f4b1d235640 1 -- 172.21.15.1:0/4119246600 shutdown_connections 2024-06-18T02:31:15.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:15.266+0000 7f4b1d235640 1 --2- 172.21.15.1:0/4119246600 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f4b04034900 0x7f4b04036db0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:31:15.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:15.266+0000 7f4b1d235640 1 --2- 172.21.15.1:0/4119246600 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b180fa370 0x7f4b180fcda0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:31:15.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:15.266+0000 7f4b1d235640 1 -- 172.21.15.1:0/4119246600 >> 172.21.15.1:0/4119246600 conn(0x7f4b180f5d70 msgr2=0x7f4b181030e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:31:15.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:15.266+0000 7f4b1d235640 1 -- 172.21.15.1:0/4119246600 shutdown_connections 2024-06-18T02:31:15.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:15.266+0000 7f4b1d235640 1 -- 172.21.15.1:0/4119246600 wait complete. 2024-06-18T02:31:15.918 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi028 2024-06-18T02:31:15.918 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-06-18T02:31:15.919 DEBUG:teuthology.orchestra.run.smithi028:> dd of=/etc/ceph/ceph.conf 2024-06-18T02:31:15.938 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-06-18T02:31:15.938 DEBUG:teuthology.orchestra.run.smithi028:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-06-18T02:31:15.997 INFO:tasks.cephadm:Adding host smithi028 to orchestrator... 2024-06-18T02:31:15.997 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph orch host add smithi028 2024-06-18T02:31:16.290 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:31:16.408 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:16 smithi001 ceph-mon[29099]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-06-18T02:31:16.408 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:16 smithi001 ceph-mon[29099]: from='mgr.14160 172.21.15.1:0/1498115122' entity='mgr.smithi001.teoawz' 2024-06-18T02:31:16.408 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:16 smithi001 ceph-mon[29099]: from='mgr.14160 172.21.15.1:0/1498115122' entity='mgr.smithi001.teoawz' 2024-06-18T02:31:16.408 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:16 smithi001 ceph-mon[29099]: from='mgr.14160 172.21.15.1:0/1498115122' entity='mgr.smithi001.teoawz' 2024-06-18T02:31:16.408 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:16 smithi001 ceph-mon[29099]: from='mgr.14160 172.21.15.1:0/1498115122' entity='mgr.smithi001.teoawz' 2024-06-18T02:31:16.408 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:16 smithi001 ceph-mon[29099]: from='mgr.14160 172.21.15.1:0/1498115122' entity='mgr.smithi001.teoawz' 2024-06-18T02:31:17.347 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:17 smithi001 ceph-mon[29099]: Deploying daemon node-exporter.smithi001 on smithi001 2024-06-18T02:31:17.348 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:17 smithi001 ceph-mon[29099]: from='mgr.14160 172.21.15.1:0/1498115122' entity='mgr.smithi001.teoawz' 2024-06-18T02:31:17.348 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:17 smithi001 ceph-mon[29099]: from='mgr.14160 172.21.15.1:0/1498115122' entity='mgr.smithi001.teoawz' 2024-06-18T02:31:17.348 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:17 smithi001 ceph-mon[29099]: from='mgr.14160 172.21.15.1:0/1498115122' entity='mgr.smithi001.teoawz' 2024-06-18T02:31:17.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:17.811+0000 7fa77ae8d640 1 -- 172.21.15.1:0/3683986469 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa774065350 msgr2=0x7fa7740fedb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:31:17.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:17.811+0000 7fa77ae8d640 1 --2- 172.21.15.1:0/3683986469 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa774065350 0x7fa7740fedb0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa768004850 tx=0x7fa768009da0 comp rx=0 tx=0).stop 2024-06-18T02:31:17.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:17.811+0000 7fa77ae8d640 1 -- 172.21.15.1:0/3683986469 shutdown_connections 2024-06-18T02:31:17.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:17.811+0000 7fa77ae8d640 1 --2- 172.21.15.1:0/3683986469 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa774065350 0x7fa7740fedb0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:31:17.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:17.811+0000 7fa77ae8d640 1 -- 172.21.15.1:0/3683986469 >> 172.21.15.1:0/3683986469 conn(0x7fa7740f6820 msgr2=0x7fa7740f8c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:31:17.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:17.811+0000 7fa77ae8d640 1 -- 172.21.15.1:0/3683986469 shutdown_connections 2024-06-18T02:31:17.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:17.811+0000 7fa77ae8d640 1 -- 172.21.15.1:0/3683986469 wait complete. 2024-06-18T02:31:17.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:17.812+0000 7fa77ae8d640 1 Processor -- start 2024-06-18T02:31:17.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:17.812+0000 7fa77ae8d640 1 -- start start 2024-06-18T02:31:17.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:17.812+0000 7fa77ae8d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa774065350 0x7fa77410ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:31:17.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:17.812+0000 7fa77ae8d640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa77410ef60 con 0x7fa774065350 2024-06-18T02:31:17.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:17.812+0000 7fa778c29640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa774065350 0x7fa77410ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:31:17.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:17.812+0000 7fa778c29640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa774065350 0x7fa77410ea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:32872/0 (socket says 172.21.15.1:32872) 2024-06-18T02:31:17.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:17.812+0000 7fa778c29640 1 -- 172.21.15.1:0/3444718021 learned_addr learned my addr 172.21.15.1:0/3444718021 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:31:17.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:17.812+0000 7fa778c29640 1 -- 172.21.15.1:0/3444718021 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa768005040 con 0x7fa774065350 2024-06-18T02:31:17.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:17.813+0000 7fa778c29640 1 --2- 172.21.15.1:0/3444718021 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa774065350 0x7fa77410ea20 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa768003080 tx=0x7fa768009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:31:17.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:17.813+0000 7fa7717fa640 1 -- 172.21.15.1:0/3444718021 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa7680326c0 con 0x7fa774065350 2024-06-18T02:31:17.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:17.813+0000 7fa77ae8d640 1 -- 172.21.15.1:0/3444718021 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa77410f160 con 0x7fa774065350 2024-06-18T02:31:17.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:17.813+0000 7fa7717fa640 1 -- 172.21.15.1:0/3444718021 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa768032cf0 con 0x7fa774065350 2024-06-18T02:31:17.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:17.813+0000 7fa7717fa640 1 -- 172.21.15.1:0/3444718021 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa768033db0 con 0x7fa774065350 2024-06-18T02:31:17.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:17.813+0000 7fa77ae8d640 1 -- 172.21.15.1:0/3444718021 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa77410fca0 con 0x7fa774065350 2024-06-18T02:31:17.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:17.814+0000 7fa7717fa640 1 -- 172.21.15.1:0/3444718021 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) v1 ==== 42001+0+0 (secure 0 0 0) 0x7fa768030020 con 0x7fa774065350 2024-06-18T02:31:17.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:17.814+0000 7fa77ae8d640 1 -- 172.21.15.1:0/3444718021 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7741026c0 con 0x7fa774065350 2024-06-18T02:31:17.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:17.814+0000 7fa7717fa640 1 --2- 172.21.15.1:0/3444718021 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7fa764034900 0x7fa764036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:31:17.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:17.814+0000 7fa7717fa640 1 -- 172.21.15.1:0/3444718021 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fa768067790 con 0x7fa774065350 2024-06-18T02:31:17.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:17.814+0000 7fa773fff640 1 --2- 172.21.15.1:0/3444718021 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7fa764034900 0x7fa764036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:31:17.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:17.815+0000 7fa773fff640 1 --2- 172.21.15.1:0/3444718021 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7fa764034900 0x7fa764036db0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fa760006fd0 tx=0x7fa760006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:31:17.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:17.819+0000 7fa7717fa640 1 -- 172.21.15.1:0/3444718021 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fa76803c380 con 0x7fa774065350 2024-06-18T02:31:18.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:18.009+0000 7fa77ae8d640 1 -- 172.21.15.1:0/3444718021 --> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi028", "target": ["mon-mgr", ""]}) v1 -- 0x7fa7740008d0 con 0x7fa764034900 2024-06-18T02:31:18.603 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:18 smithi001 ceph-mon[29099]: from='client.14186 -' entity='client.crash.smithi001' cmd=[{"prefix": "crash post", "target": ["mon-mgr", ""]}]: dispatch 2024-06-18T02:31:18.603 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:18 smithi001 ceph-mon[29099]: from='client.14188 -' entity='client.crash.smithi001' cmd=[{"prefix": "crash post", "target": ["mon-mgr", ""]}]: dispatch 2024-06-18T02:31:18.603 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:18 smithi001 ceph-mon[29099]: from='client.14190 -' entity='client.crash.smithi001' cmd=[{"prefix": "crash post", "target": ["mon-mgr", ""]}]: dispatch 2024-06-18T02:31:19.627 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:19 smithi001 ceph-mon[29099]: from='client.14192 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi028", "target": ["mon-mgr", ""]}]: dispatch 2024-06-18T02:31:19.628 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:19 smithi001 ceph-mon[29099]: from='mgr.14160 172.21.15.1:0/1498115122' entity='mgr.smithi001.teoawz' 2024-06-18T02:31:20.561 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:20 smithi001 ceph-mon[29099]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:31:20.562 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:20 smithi001 ceph-mon[29099]: Deploying cephadm binary to smithi028 2024-06-18T02:31:20.562 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:20 smithi001 ceph-mon[29099]: Health check failed: 3 mgr modules have recently crashed (RECENT_MGR_MODULE_CRASH) 2024-06-18T02:31:21.567 INFO:teuthology.orchestra.run.smithi001.stdout:Added host 'smithi028' with addr '172.21.15.28' 2024-06-18T02:31:21.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:21.565+0000 7fa7717fa640 1 -- 172.21.15.1:0/3444718021 <== mgr.14160 v2:172.21.15.1:6800/85550937 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7fa7740008d0 con 0x7fa764034900 2024-06-18T02:31:21.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:21.568+0000 7fa77ae8d640 1 -- 172.21.15.1:0/3444718021 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7fa764034900 msgr2=0x7fa764036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:31:21.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:21.568+0000 7fa77ae8d640 1 --2- 172.21.15.1:0/3444718021 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7fa764034900 0x7fa764036db0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fa760006fd0 tx=0x7fa760006db0 comp rx=0 tx=0).stop 2024-06-18T02:31:21.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:21.568+0000 7fa77ae8d640 1 -- 172.21.15.1:0/3444718021 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa774065350 msgr2=0x7fa77410ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:31:21.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:21.568+0000 7fa77ae8d640 1 --2- 172.21.15.1:0/3444718021 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa774065350 0x7fa77410ea20 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa768003080 tx=0x7fa768009da0 comp rx=0 tx=0).stop 2024-06-18T02:31:21.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:21.568+0000 7fa77ae8d640 1 -- 172.21.15.1:0/3444718021 shutdown_connections 2024-06-18T02:31:21.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:21.568+0000 7fa77ae8d640 1 --2- 172.21.15.1:0/3444718021 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7fa764034900 0x7fa764036db0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:31:21.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:21.568+0000 7fa77ae8d640 1 --2- 172.21.15.1:0/3444718021 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa774065350 0x7fa77410ea20 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:31:21.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:21.568+0000 7fa77ae8d640 1 -- 172.21.15.1:0/3444718021 >> 172.21.15.1:0/3444718021 conn(0x7fa7740f6820 msgr2=0x7fa7740f8c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:31:21.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:21.568+0000 7fa77ae8d640 1 -- 172.21.15.1:0/3444718021 shutdown_connections 2024-06-18T02:31:21.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:21.568+0000 7fa77ae8d640 1 -- 172.21.15.1:0/3444718021 wait complete. 2024-06-18T02:31:22.343 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:21 smithi001 ceph-mon[29099]: from='mgr.14160 172.21.15.1:0/1498115122' entity='mgr.smithi001.teoawz' 2024-06-18T02:31:22.343 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:21 smithi001 ceph-mon[29099]: from='mgr.14160 172.21.15.1:0/1498115122' entity='mgr.smithi001.teoawz' 2024-06-18T02:31:22.343 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:21 smithi001 ceph-mon[29099]: from='mgr.14160 172.21.15.1:0/1498115122' entity='mgr.smithi001.teoawz' 2024-06-18T02:31:22.343 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:21 smithi001 ceph-mon[29099]: from='mgr.14160 172.21.15.1:0/1498115122' entity='mgr.smithi001.teoawz' 2024-06-18T02:31:22.343 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:21 smithi001 ceph-mon[29099]: Deploying daemon alertmanager.smithi001 on smithi001 2024-06-18T02:31:22.343 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:21 smithi001 ceph-mon[29099]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:31:22.343 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:21 smithi001 ceph-mon[29099]: from='mgr.14160 172.21.15.1:0/1498115122' entity='mgr.smithi001.teoawz' 2024-06-18T02:31:22.368 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph orch host ls --format=json 2024-06-18T02:31:22.627 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:31:23.082 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:22 smithi001 ceph-mon[29099]: Added host smithi028 2024-06-18T02:31:24.377 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:23 smithi001 ceph-mon[29099]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:31:25.377 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:25 smithi001 ceph-mon[29099]: from='mgr.14160 172.21.15.1:0/1498115122' entity='mgr.smithi001.teoawz' 2024-06-18T02:31:25.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:25.872+0000 7fd8e5d6b640 1 -- 172.21.15.1:0/1811484010 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8e00712b0 msgr2=0x7fd8e006f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:31:25.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:25.872+0000 7fd8e5d6b640 1 --2- 172.21.15.1:0/1811484010 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8e00712b0 0x7fd8e006f7e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd8d0009bd0 tx=0x7fd8d0004d10 comp rx=0 tx=0).stop 2024-06-18T02:31:25.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:25.872+0000 7fd8e5d6b640 1 -- 172.21.15.1:0/1811484010 shutdown_connections 2024-06-18T02:31:25.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:25.872+0000 7fd8e5d6b640 1 --2- 172.21.15.1:0/1811484010 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8e00712b0 0x7fd8e006f7e0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:31:25.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:25.872+0000 7fd8e5d6b640 1 -- 172.21.15.1:0/1811484010 >> 172.21.15.1:0/1811484010 conn(0x7fd8e00f76f0 msgr2=0x7fd8e00f9b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:31:25.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:25.872+0000 7fd8e5d6b640 1 -- 172.21.15.1:0/1811484010 shutdown_connections 2024-06-18T02:31:25.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:25.872+0000 7fd8e5d6b640 1 -- 172.21.15.1:0/1811484010 wait complete. 2024-06-18T02:31:25.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:25.872+0000 7fd8e5d6b640 1 Processor -- start 2024-06-18T02:31:25.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:25.873+0000 7fd8e5d6b640 1 -- start start 2024-06-18T02:31:25.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:25.873+0000 7fd8e5d6b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8e00712b0 0x7fd8e010e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:31:25.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:25.873+0000 7fd8e5d6b640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e010ecc0 con 0x7fd8e00712b0 2024-06-18T02:31:25.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:25.873+0000 7fd8df7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8e00712b0 0x7fd8e010e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:31:25.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:25.873+0000 7fd8df7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8e00712b0 0x7fd8e010e780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41074/0 (socket says 172.21.15.1:41074) 2024-06-18T02:31:25.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:25.873+0000 7fd8df7fe640 1 -- 172.21.15.1:0/3111116014 learned_addr learned my addr 172.21.15.1:0/3111116014 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:31:25.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:25.873+0000 7fd8df7fe640 1 -- 172.21.15.1:0/3111116014 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8d00098b0 con 0x7fd8e00712b0 2024-06-18T02:31:25.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:25.873+0000 7fd8df7fe640 1 --2- 172.21.15.1:0/3111116014 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8e00712b0 0x7fd8e010e780 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd8d0009ba0 tx=0x7fd8d0004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:31:25.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:25.873+0000 7fd8c7fff640 1 -- 172.21.15.1:0/3111116014 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd8d00329b0 con 0x7fd8e00712b0 2024-06-18T02:31:25.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:25.874+0000 7fd8e5d6b640 1 -- 172.21.15.1:0/3111116014 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8e010eec0 con 0x7fd8e00712b0 2024-06-18T02:31:25.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:25.874+0000 7fd8c7fff640 1 -- 172.21.15.1:0/3111116014 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd8d003d420 con 0x7fd8e00712b0 2024-06-18T02:31:25.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:25.874+0000 7fd8c7fff640 1 -- 172.21.15.1:0/3111116014 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd8d0031590 con 0x7fd8e00712b0 2024-06-18T02:31:25.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:25.874+0000 7fd8e5d6b640 1 -- 172.21.15.1:0/3111116014 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8e010f310 con 0x7fd8e00712b0 2024-06-18T02:31:25.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:25.875+0000 7fd8c7fff640 1 -- 172.21.15.1:0/3111116014 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) v1 ==== 42001+0+0 (secure 0 0 0) 0x7fd8d002f030 con 0x7fd8e00712b0 2024-06-18T02:31:25.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:25.875+0000 7fd8e5d6b640 1 -- 172.21.15.1:0/3111116014 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8e005ead0 con 0x7fd8e00712b0 2024-06-18T02:31:25.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:25.875+0000 7fd8c7fff640 1 --2- 172.21.15.1:0/3111116014 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7fd8c80348b0 0x7fd8c8036d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:31:25.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:25.875+0000 7fd8c7fff640 1 -- 172.21.15.1:0/3111116014 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fd8d0067c50 con 0x7fd8e00712b0 2024-06-18T02:31:25.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:25.875+0000 7fd8deffd640 1 --2- 172.21.15.1:0/3111116014 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7fd8c80348b0 0x7fd8c8036d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:31:25.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:25.875+0000 7fd8deffd640 1 --2- 172.21.15.1:0/3111116014 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7fd8c80348b0 0x7fd8c8036d60 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fd8d4006fd0 tx=0x7fd8d4006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:31:25.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:25.879+0000 7fd8c7fff640 1 -- 172.21.15.1:0/3111116014 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fd8d006d020 con 0x7fd8e00712b0 2024-06-18T02:31:26.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:26.058+0000 7fd8e5d6b640 1 -- 172.21.15.1:0/3111116014 --> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd8e0110340 con 0x7fd8c80348b0 2024-06-18T02:31:26.059 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:26 smithi001 ceph-mon[29099]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:31:26.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:26.059+0000 7fd8c7fff640 1 -- 172.21.15.1:0/3111116014 <== mgr.14160 v2:172.21.15.1:6800/85550937 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+158 (secure 0 0 0) 0x7fd8e0110340 con 0x7fd8c80348b0 2024-06-18T02:31:26.059 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:31:26.059 INFO:teuthology.orchestra.run.smithi001.stdout:[{"addr": "172.21.15.1", "hostname": "smithi001", "labels": [], "status": ""}, {"addr": "172.21.15.28", "hostname": "smithi028", "labels": [], "status": ""}] 2024-06-18T02:31:26.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:26.061+0000 7fd8e5d6b640 1 -- 172.21.15.1:0/3111116014 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7fd8c80348b0 msgr2=0x7fd8c8036d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:31:26.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:26.061+0000 7fd8e5d6b640 1 --2- 172.21.15.1:0/3111116014 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7fd8c80348b0 0x7fd8c8036d60 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fd8d4006fd0 tx=0x7fd8d4006db0 comp rx=0 tx=0).stop 2024-06-18T02:31:26.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:26.061+0000 7fd8e5d6b640 1 -- 172.21.15.1:0/3111116014 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8e00712b0 msgr2=0x7fd8e010e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:31:26.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:26.061+0000 7fd8e5d6b640 1 --2- 172.21.15.1:0/3111116014 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8e00712b0 0x7fd8e010e780 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd8d0009ba0 tx=0x7fd8d0004d10 comp rx=0 tx=0).stop 2024-06-18T02:31:26.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:26.061+0000 7fd8e5d6b640 1 -- 172.21.15.1:0/3111116014 shutdown_connections 2024-06-18T02:31:26.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:26.061+0000 7fd8e5d6b640 1 --2- 172.21.15.1:0/3111116014 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7fd8c80348b0 0x7fd8c8036d60 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:31:26.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:26.061+0000 7fd8e5d6b640 1 --2- 172.21.15.1:0/3111116014 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8e00712b0 0x7fd8e010e780 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:31:26.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:26.061+0000 7fd8e5d6b640 1 -- 172.21.15.1:0/3111116014 >> 172.21.15.1:0/3111116014 conn(0x7fd8e00f76f0 msgr2=0x7fd8e01052b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:31:26.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:26.061+0000 7fd8e5d6b640 1 -- 172.21.15.1:0/3111116014 shutdown_connections 2024-06-18T02:31:26.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:26.061+0000 7fd8e5d6b640 1 -- 172.21.15.1:0/3111116014 wait complete. 2024-06-18T02:31:27.213 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:27 smithi001 ceph-mon[29099]: from='client.14194 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-18T02:31:27.245 INFO:tasks.cephadm:Setting crush tunables to default 2024-06-18T02:31:27.245 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph osd crush tunables default 2024-06-18T02:31:28.127 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:28 smithi001 ceph-mon[29099]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:31:29.522 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:31:30.377 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:30 smithi001 ceph-mon[29099]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:31:31.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:31.114+0000 7f26144a6640 1 -- 172.21.15.1:0/2219554164 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f260c065bc0 msgr2=0x7f260c100fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:31:31.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:31.114+0000 7f26144a6640 1 --2- 172.21.15.1:0/2219554164 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f260c065bc0 0x7f260c100fb0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2608004850 tx=0x7f2608009da0 comp rx=0 tx=0).stop 2024-06-18T02:31:31.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:31.114+0000 7f26144a6640 1 -- 172.21.15.1:0/2219554164 shutdown_connections 2024-06-18T02:31:31.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:31.114+0000 7f26144a6640 1 --2- 172.21.15.1:0/2219554164 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f260c065bc0 0x7f260c100fb0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:31:31.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:31.114+0000 7f26144a6640 1 -- 172.21.15.1:0/2219554164 >> 172.21.15.1:0/2219554164 conn(0x7f260c0f65e0 msgr2=0x7f260c0f8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:31:31.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:31.114+0000 7f26144a6640 1 -- 172.21.15.1:0/2219554164 shutdown_connections 2024-06-18T02:31:31.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:31.114+0000 7f26144a6640 1 -- 172.21.15.1:0/2219554164 wait complete. 2024-06-18T02:31:31.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:31.114+0000 7f26144a6640 1 Processor -- start 2024-06-18T02:31:31.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:31.114+0000 7f26144a6640 1 -- start start 2024-06-18T02:31:31.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:31.115+0000 7f26144a6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f260c065bc0 0x7f260c10e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:31:31.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:31.115+0000 7f26144a6640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f260c10ecd0 con 0x7f260c065bc0 2024-06-18T02:31:31.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:31.115+0000 7f2612242640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f260c065bc0 0x7f260c10e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:31:31.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:31.115+0000 7f2612242640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f260c065bc0 0x7f260c10e790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41082/0 (socket says 172.21.15.1:41082) 2024-06-18T02:31:31.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:31.115+0000 7f2612242640 1 -- 172.21.15.1:0/1518994594 learned_addr learned my addr 172.21.15.1:0/1518994594 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:31:31.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:31.115+0000 7f2612242640 1 -- 172.21.15.1:0/1518994594 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2608005040 con 0x7f260c065bc0 2024-06-18T02:31:31.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:31.115+0000 7f2612242640 1 --2- 172.21.15.1:0/1518994594 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f260c065bc0 0x7f260c10e790 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2608004820 tx=0x7f2608009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:31:31.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:31.115+0000 7f2602ffd640 1 -- 172.21.15.1:0/1518994594 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2608032650 con 0x7f260c065bc0 2024-06-18T02:31:31.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:31.115+0000 7f26144a6640 1 -- 172.21.15.1:0/1518994594 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f260c10eed0 con 0x7f260c065bc0 2024-06-18T02:31:31.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:31.115+0000 7f2602ffd640 1 -- 172.21.15.1:0/1518994594 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2608032c80 con 0x7f260c065bc0 2024-06-18T02:31:31.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:31.115+0000 7f2602ffd640 1 -- 172.21.15.1:0/1518994594 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2608033d60 con 0x7f260c065bc0 2024-06-18T02:31:31.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:31.115+0000 7f26144a6640 1 -- 172.21.15.1:0/1518994594 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f260c10fa60 con 0x7f260c065bc0 2024-06-18T02:31:31.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:31.116+0000 7f26144a6640 1 -- 172.21.15.1:0/1518994594 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f260c10f3a0 con 0x7f260c065bc0 2024-06-18T02:31:31.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:31.116+0000 7f2602ffd640 1 -- 172.21.15.1:0/1518994594 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) v1 ==== 42001+0+0 (secure 0 0 0) 0x7f2608030020 con 0x7f260c065bc0 2024-06-18T02:31:31.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:31.117+0000 7f2602ffd640 1 --2- 172.21.15.1:0/1518994594 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f25f8034900 0x7f25f8036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:31:31.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:31.117+0000 7f2602ffd640 1 -- 172.21.15.1:0/1518994594 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f2608066710 con 0x7f260c065bc0 2024-06-18T02:31:31.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:31.117+0000 7f2611a41640 1 --2- 172.21.15.1:0/1518994594 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f25f8034900 0x7f25f8036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:31:31.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:31.117+0000 7f2611a41640 1 --2- 172.21.15.1:0/1518994594 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f25f8034900 0x7f25f8036db0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f25fc006fd0 tx=0x7f25fc006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:31:31.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:31.125+0000 7f2602ffd640 1 -- 172.21.15.1:0/1518994594 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f260c10f3a0 con 0x7f260c065bc0 2024-06-18T02:31:31.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:31.311+0000 7f26144a6640 1 -- 172.21.15.1:0/1518994594 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f260c104690 con 0x7f260c065bc0 2024-06-18T02:31:32.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:32.038+0000 7f2602ffd640 1 -- 172.21.15.1:0/1518994594 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f26080453a0 con 0x7f260c065bc0 2024-06-18T02:31:32.039 INFO:teuthology.orchestra.run.smithi001.stderr:adjusted tunables profile to default 2024-06-18T02:31:32.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:32.041+0000 7f26144a6640 1 -- 172.21.15.1:0/1518994594 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f25f8034900 msgr2=0x7f25f8036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:31:32.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:32.041+0000 7f26144a6640 1 --2- 172.21.15.1:0/1518994594 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f25f8034900 0x7f25f8036db0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f25fc006fd0 tx=0x7f25fc006db0 comp rx=0 tx=0).stop 2024-06-18T02:31:32.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:32.041+0000 7f26144a6640 1 -- 172.21.15.1:0/1518994594 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f260c065bc0 msgr2=0x7f260c10e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:31:32.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:32.041+0000 7f26144a6640 1 --2- 172.21.15.1:0/1518994594 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f260c065bc0 0x7f260c10e790 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2608004820 tx=0x7f2608009da0 comp rx=0 tx=0).stop 2024-06-18T02:31:32.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:32.041+0000 7f26144a6640 1 -- 172.21.15.1:0/1518994594 shutdown_connections 2024-06-18T02:31:32.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:32.041+0000 7f26144a6640 1 --2- 172.21.15.1:0/1518994594 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f25f8034900 0x7f25f8036db0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:31:32.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:32.041+0000 7f26144a6640 1 --2- 172.21.15.1:0/1518994594 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f260c065bc0 0x7f260c10e790 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:31:32.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:32.041+0000 7f26144a6640 1 -- 172.21.15.1:0/1518994594 >> 172.21.15.1:0/1518994594 conn(0x7f260c0f65e0 msgr2=0x7f260c1052c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:31:32.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:32.041+0000 7f26144a6640 1 -- 172.21.15.1:0/1518994594 shutdown_connections 2024-06-18T02:31:32.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:31:32.041+0000 7f26144a6640 1 -- 172.21.15.1:0/1518994594 wait complete. 2024-06-18T02:31:32.377 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:32 smithi001 ceph-mon[29099]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:31:32.377 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:32 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/1518994594' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-06-18T02:31:32.765 INFO:tasks.cephadm:Adding mon.smithi001 on smithi001 2024-06-18T02:31:32.765 INFO:tasks.cephadm:Adding mon.smithi028 on smithi028 2024-06-18T02:31:32.766 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph orch apply mon '2;smithi001:172.21.15.1=smithi001;smithi028:172.21.15.28=smithi028' 2024-06-18T02:31:33.029 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-18T02:31:33.076 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-18T02:31:33.377 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:33 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/1518994594' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-06-18T02:31:33.377 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:33 smithi001 ceph-mon[29099]: osdmap e4: 0 total, 0 up, 0 in 2024-06-18T02:31:33.377 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:33 smithi001 ceph-mon[29099]: from='mgr.14160 172.21.15.1:0/1498115122' entity='mgr.smithi001.teoawz' 2024-06-18T02:31:33.377 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:33 smithi001 ceph-mon[29099]: from='mgr.14160 172.21.15.1:0/1498115122' entity='mgr.smithi001.teoawz' 2024-06-18T02:31:33.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:33 smithi001 ceph-mon[29099]: from='mgr.14160 172.21.15.1:0/1498115122' entity='mgr.smithi001.teoawz' 2024-06-18T02:31:33.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:33 smithi001 ceph-mon[29099]: from='mgr.14160 172.21.15.1:0/1498115122' entity='mgr.smithi001.teoawz' 2024-06-18T02:31:33.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:33 smithi001 ceph-mon[29099]: from='mgr.14160 172.21.15.1:0/1498115122' entity='mgr.smithi001.teoawz' 2024-06-18T02:31:33.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:33 smithi001 ceph-mon[29099]: from='mgr.14160 172.21.15.1:0/1498115122' entity='mgr.smithi001.teoawz' 2024-06-18T02:31:33.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:33 smithi001 ceph-mon[29099]: from='mgr.14160 172.21.15.1:0/1498115122' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-06-18T02:31:33.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:33 smithi001 ceph-mon[29099]: from='mgr.14160 172.21.15.1:0/1498115122' entity='mgr.smithi001.teoawz' 2024-06-18T02:31:34.377 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:34 smithi001 ceph-mon[29099]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-06-18T02:31:34.377 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:34 smithi001 ceph-mon[29099]: Deploying daemon grafana.smithi001 on smithi001 2024-06-18T02:31:34.377 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:34 smithi001 ceph-mon[29099]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:31:34.377 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:34 smithi001 ceph-mon[29099]: from='mgr.14160 172.21.15.1:0/1498115122' entity='mgr.smithi001.teoawz' 2024-06-18T02:31:34.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:34.590+0000 7fb80edca640 1 -- 172.21.15.28:0/3996749920 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8080fa310 msgr2=0x7fb8080fa720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:31:34.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:34.590+0000 7fb80edca640 1 --2- 172.21.15.28:0/3996749920 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8080fa310 0x7fb8080fa720 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8004850 tx=0x7fb7f8009da0 comp rx=0 tx=0).stop 2024-06-18T02:31:34.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:34.590+0000 7fb80edca640 1 -- 172.21.15.28:0/3996749920 shutdown_connections 2024-06-18T02:31:34.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:34.590+0000 7fb80edca640 1 --2- 172.21.15.28:0/3996749920 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8080fa310 0x7fb8080fa720 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:31:34.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:34.590+0000 7fb80edca640 1 -- 172.21.15.28:0/3996749920 >> 172.21.15.28:0/3996749920 conn(0x7fb8080f5d70 msgr2=0x7fb8080f8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:31:34.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:34.590+0000 7fb80edca640 1 -- 172.21.15.28:0/3996749920 shutdown_connections 2024-06-18T02:31:34.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:34.590+0000 7fb80edca640 1 -- 172.21.15.28:0/3996749920 wait complete. 2024-06-18T02:31:34.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:34.591+0000 7fb80edca640 1 Processor -- start 2024-06-18T02:31:34.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:34.591+0000 7fb80edca640 1 -- start start 2024-06-18T02:31:34.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:34.591+0000 7fb80edca640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8080fa310 0x7fb80810c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:31:34.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:34.591+0000 7fb80edca640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb80810ccf0 con 0x7fb8080fa310 2024-06-18T02:31:34.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:34.591+0000 7fb80cb66640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8080fa310 0x7fb80810c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:31:34.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:34.591+0000 7fb80cb66640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8080fa310 0x7fb80810c7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.28:44048/0 (socket says 172.21.15.28:44048) 2024-06-18T02:31:34.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:34.591+0000 7fb80cb66640 1 -- 172.21.15.28:0/3748080572 learned_addr learned my addr 172.21.15.28:0/3748080572 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-06-18T02:31:34.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:34.592+0000 7fb80cb66640 1 -- 172.21.15.28:0/3748080572 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7f8005040 con 0x7fb8080fa310 2024-06-18T02:31:34.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:34.592+0000 7fb80cb66640 1 --2- 172.21.15.28:0/3748080572 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8080fa310 0x7fb80810c7b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8004820 tx=0x7fb7f8009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:31:34.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:34.592+0000 7fb8057fa640 1 -- 172.21.15.28:0/3748080572 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7f8032650 con 0x7fb8080fa310 2024-06-18T02:31:34.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:34.592+0000 7fb8057fa640 1 -- 172.21.15.28:0/3748080572 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f8032c80 con 0x7fb8080fa310 2024-06-18T02:31:34.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:34.592+0000 7fb80edca640 1 -- 172.21.15.28:0/3748080572 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb80810cef0 con 0x7fb8080fa310 2024-06-18T02:31:34.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:34.592+0000 7fb8057fa640 1 -- 172.21.15.28:0/3748080572 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7f803eda0 con 0x7fb8080fa310 2024-06-18T02:31:34.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:34.592+0000 7fb80edca640 1 -- 172.21.15.28:0/3748080572 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb80810da80 con 0x7fb8080fa310 2024-06-18T02:31:34.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:34.593+0000 7fb8057fa640 1 -- 172.21.15.28:0/3748080572 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) v1 ==== 42001+0+0 (secure 0 0 0) 0x7fb7f8030020 con 0x7fb8080fa310 2024-06-18T02:31:34.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:34.593+0000 7fb80edca640 1 -- 172.21.15.28:0/3748080572 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8081026b0 con 0x7fb8080fa310 2024-06-18T02:31:34.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:34.593+0000 7fb8057fa640 1 --2- 172.21.15.28:0/3748080572 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7fb7f0034950 0x7fb7f0036e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:31:34.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:34.593+0000 7fb8057fa640 1 -- 172.21.15.28:0/3748080572 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fb7f80669b0 con 0x7fb8080fa310 2024-06-18T02:31:34.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:34.594+0000 7fb807fff640 1 --2- 172.21.15.28:0/3748080572 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7fb7f0034950 0x7fb7f0036e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:31:34.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:34.594+0000 7fb807fff640 1 --2- 172.21.15.28:0/3748080572 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7fb7f0034950 0x7fb7f0036e00 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc006fd0 tx=0x7fb7fc006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:31:34.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:34.599+0000 7fb8057fa640 1 -- 172.21.15.28:0/3748080572 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fb7f803c510 con 0x7fb8080fa310 2024-06-18T02:31:34.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:34.781+0000 7fb80edca640 1 -- 172.21.15.28:0/3748080572 --> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi001:172.21.15.1=smithi001;smithi028:172.21.15.28=smithi028", "target": ["mon-mgr", ""]}) v1 -- 0x7fb808000980 con 0x7fb7f0034950 2024-06-18T02:31:34.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:34.787+0000 7fb8057fa640 1 -- 172.21.15.28:0/3748080572 <== mgr.14160 v2:172.21.15.1:6800/85550937 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fb808000980 con 0x7fb7f0034950 2024-06-18T02:31:34.786 INFO:teuthology.orchestra.run.smithi028.stdout:Scheduled mon update... 2024-06-18T02:31:34.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:34.788+0000 7fb80edca640 1 -- 172.21.15.28:0/3748080572 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7fb7f0034950 msgr2=0x7fb7f0036e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:31:34.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:34.788+0000 7fb80edca640 1 --2- 172.21.15.28:0/3748080572 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7fb7f0034950 0x7fb7f0036e00 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc006fd0 tx=0x7fb7fc006db0 comp rx=0 tx=0).stop 2024-06-18T02:31:34.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:34.788+0000 7fb80edca640 1 -- 172.21.15.28:0/3748080572 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8080fa310 msgr2=0x7fb80810c7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:31:34.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:34.788+0000 7fb80edca640 1 --2- 172.21.15.28:0/3748080572 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8080fa310 0x7fb80810c7b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8004820 tx=0x7fb7f8009da0 comp rx=0 tx=0).stop 2024-06-18T02:31:34.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:34.788+0000 7fb80edca640 1 -- 172.21.15.28:0/3748080572 shutdown_connections 2024-06-18T02:31:34.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:34.788+0000 7fb80edca640 1 --2- 172.21.15.28:0/3748080572 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7fb7f0034950 0x7fb7f0036e00 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:31:34.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:34.788+0000 7fb80edca640 1 --2- 172.21.15.28:0/3748080572 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8080fa310 0x7fb80810c7b0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:31:34.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:34.788+0000 7fb80edca640 1 -- 172.21.15.28:0/3748080572 >> 172.21.15.28:0/3748080572 conn(0x7fb8080f5d70 msgr2=0x7fb8081032e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:31:34.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:34.788+0000 7fb80edca640 1 -- 172.21.15.28:0/3748080572 shutdown_connections 2024-06-18T02:31:34.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:34.788+0000 7fb80edca640 1 -- 172.21.15.28:0/3748080572 wait complete. 2024-06-18T02:31:35.353 DEBUG:teuthology.orchestra.run.smithi028:mon.smithi028> sudo journalctl -f -n 0 -u ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968@mon.smithi028.service 2024-06-18T02:31:35.356 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-18T02:31:35.356 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph mon dump -f json 2024-06-18T02:31:35.644 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-18T02:31:35.693 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-18T02:31:36.127 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:35 smithi001 ceph-mon[29099]: from='client.14198 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi001:172.21.15.1=smithi001;smithi028:172.21.15.28=smithi028", "target": ["mon-mgr", ""]}]: dispatch 2024-06-18T02:31:36.127 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:35 smithi001 ceph-mon[29099]: Saving service mon spec with placement smithi001:172.21.15.1=smithi001;smithi028:172.21.15.28=smithi028;count:2 2024-06-18T02:31:36.127 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:35 smithi001 ceph-mon[29099]: from='mgr.14160 172.21.15.1:0/1498115122' entity='mgr.smithi001.teoawz' 2024-06-18T02:31:36.127 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:35 smithi001 ceph-mon[29099]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:31:37.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:37.162+0000 7f826e8e6640 1 -- 172.21.15.28:0/3580053144 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82680fb430 msgr2=0x7f82680fb840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:31:37.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:37.162+0000 7f826e8e6640 1 --2- 172.21.15.28:0/3580053144 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82680fb430 0x7f82680fb840 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f8258004850 tx=0x7f8258009da0 comp rx=0 tx=0).stop 2024-06-18T02:31:37.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:37.162+0000 7f826e8e6640 1 -- 172.21.15.28:0/3580053144 shutdown_connections 2024-06-18T02:31:37.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:37.162+0000 7f826e8e6640 1 --2- 172.21.15.28:0/3580053144 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82680fb430 0x7f82680fb840 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:31:37.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:37.162+0000 7f826e8e6640 1 -- 172.21.15.28:0/3580053144 >> 172.21.15.28:0/3580053144 conn(0x7f82680f5da0 msgr2=0x7f82680f81b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:31:37.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:37.162+0000 7f826e8e6640 1 -- 172.21.15.28:0/3580053144 shutdown_connections 2024-06-18T02:31:37.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:37.162+0000 7f826e8e6640 1 -- 172.21.15.28:0/3580053144 wait complete. 2024-06-18T02:31:37.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:37.163+0000 7f826e8e6640 1 Processor -- start 2024-06-18T02:31:37.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:37.163+0000 7f826e8e6640 1 -- start start 2024-06-18T02:31:37.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:37.163+0000 7f826e8e6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82680fb430 0x7f826810c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:31:37.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:37.163+0000 7f826e8e6640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f826810cac0 con 0x7f82680fb430 2024-06-18T02:31:37.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:37.163+0000 7f8267fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82680fb430 0x7f826810c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:31:37.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:37.163+0000 7f8267fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82680fb430 0x7f826810c580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.28:44064/0 (socket says 172.21.15.28:44064) 2024-06-18T02:31:37.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:37.163+0000 7f8267fff640 1 -- 172.21.15.28:0/3181014256 learned_addr learned my addr 172.21.15.28:0/3181014256 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-06-18T02:31:37.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:37.164+0000 7f8267fff640 1 -- 172.21.15.28:0/3181014256 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8258005040 con 0x7f82680fb430 2024-06-18T02:31:37.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:37.164+0000 7f8267fff640 1 --2- 172.21.15.28:0/3181014256 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82680fb430 0x7f826810c580 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8258004820 tx=0x7f8258009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:31:37.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:37.164+0000 7f8264ff9640 1 -- 172.21.15.28:0/3181014256 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8258032650 con 0x7f82680fb430 2024-06-18T02:31:37.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:37.164+0000 7f8264ff9640 1 -- 172.21.15.28:0/3181014256 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8258032c80 con 0x7f82680fb430 2024-06-18T02:31:37.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:37.164+0000 7f826e8e6640 1 -- 172.21.15.28:0/3181014256 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f826810ccc0 con 0x7f82680fb430 2024-06-18T02:31:37.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:37.164+0000 7f8264ff9640 1 -- 172.21.15.28:0/3181014256 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8258033d60 con 0x7f82680fb430 2024-06-18T02:31:37.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:37.164+0000 7f826e8e6640 1 -- 172.21.15.28:0/3181014256 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f826810d850 con 0x7f82680fb430 2024-06-18T02:31:37.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:37.165+0000 7f8264ff9640 1 -- 172.21.15.28:0/3181014256 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) v1 ==== 42001+0+0 (secure 0 0 0) 0x7f8258030020 con 0x7f82680fb430 2024-06-18T02:31:37.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:37.165+0000 7f826e8e6640 1 -- 172.21.15.28:0/3181014256 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82681013b0 con 0x7f82680fb430 2024-06-18T02:31:37.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:37.165+0000 7f8264ff9640 1 --2- 172.21.15.28:0/3181014256 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f8250034900 0x7f8250036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:31:37.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:37.165+0000 7f8264ff9640 1 -- 172.21.15.28:0/3181014256 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f8258066800 con 0x7f82680fb430 2024-06-18T02:31:37.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:37.165+0000 7f82677fe640 1 --2- 172.21.15.28:0/3181014256 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f8250034900 0x7f8250036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:31:37.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:37.166+0000 7f82677fe640 1 --2- 172.21.15.28:0/3181014256 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f8250034900 0x7f8250036db0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f825c006fd0 tx=0x7f825c006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:31:37.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:37.170+0000 7f8264ff9640 1 -- 172.21.15.28:0/3181014256 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f82580453a0 con 0x7f82680fb430 2024-06-18T02:31:37.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:37.411+0000 7f826e8e6640 1 -- 172.21.15.28:0/3181014256 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f826810da90 con 0x7f82680fb430 2024-06-18T02:31:37.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:37.411+0000 7f8264ff9640 1 -- 172.21.15.28:0/3181014256 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+735 (secure 0 0 0) 0x7f825802c110 con 0x7f82680fb430 2024-06-18T02:31:37.411 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:31:37.412 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":1,"fsid":"7eefb63e-2d1a-11ef-bc9d-c7b262605968","modified":"2024-06-18T02:29:55.463546Z","created":"2024-06-18T02:29:55.463546Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi001","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-18T02:31:37.412 INFO:teuthology.orchestra.run.smithi028.stderr:dumped monmap epoch 1 2024-06-18T02:31:37.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:37.414+0000 7f826e8e6640 1 -- 172.21.15.28:0/3181014256 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f8250034900 msgr2=0x7f8250036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:31:37.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:37.414+0000 7f826e8e6640 1 --2- 172.21.15.28:0/3181014256 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f8250034900 0x7f8250036db0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f825c006fd0 tx=0x7f825c006db0 comp rx=0 tx=0).stop 2024-06-18T02:31:37.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:37.414+0000 7f826e8e6640 1 -- 172.21.15.28:0/3181014256 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82680fb430 msgr2=0x7f826810c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:31:37.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:37.414+0000 7f826e8e6640 1 --2- 172.21.15.28:0/3181014256 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82680fb430 0x7f826810c580 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8258004820 tx=0x7f8258009da0 comp rx=0 tx=0).stop 2024-06-18T02:31:37.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:37.414+0000 7f826e8e6640 1 -- 172.21.15.28:0/3181014256 shutdown_connections 2024-06-18T02:31:37.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:37.414+0000 7f826e8e6640 1 --2- 172.21.15.28:0/3181014256 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f8250034900 0x7f8250036db0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:31:37.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:37.414+0000 7f826e8e6640 1 --2- 172.21.15.28:0/3181014256 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82680fb430 0x7f826810c580 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:31:37.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:37.414+0000 7f826e8e6640 1 -- 172.21.15.28:0/3181014256 >> 172.21.15.28:0/3181014256 conn(0x7f82680f5da0 msgr2=0x7f82680fa0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:31:37.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:37.414+0000 7f826e8e6640 1 -- 172.21.15.28:0/3181014256 shutdown_connections 2024-06-18T02:31:37.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:37.414+0000 7f826e8e6640 1 -- 172.21.15.28:0/3181014256 wait complete. 2024-06-18T02:31:38.377 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:38 smithi001 ceph-mon[29099]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:31:38.377 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:38 smithi001 ceph-mon[29099]: from='client.? 172.21.15.28:0/3181014256' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-18T02:31:38.966 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-18T02:31:38.967 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph mon dump -f json 2024-06-18T02:31:39.227 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-18T02:31:39.275 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-18T02:31:40.377 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:40 smithi001 ceph-mon[29099]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:31:40.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:40.760+0000 7fcda32a7640 1 -- 172.21.15.28:0/3129005090 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd9c0fae70 msgr2=0x7fcd9c0fb280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:31:40.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:40.760+0000 7fcda32a7640 1 --2- 172.21.15.28:0/3129005090 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd9c0fae70 0x7fcd9c0fb280 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fcd98004850 tx=0x7fcd98009da0 comp rx=0 tx=0).stop 2024-06-18T02:31:40.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:40.760+0000 7fcda32a7640 1 -- 172.21.15.28:0/3129005090 shutdown_connections 2024-06-18T02:31:40.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:40.760+0000 7fcda32a7640 1 --2- 172.21.15.28:0/3129005090 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd9c0fae70 0x7fcd9c0fb280 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:31:40.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:40.760+0000 7fcda32a7640 1 -- 172.21.15.28:0/3129005090 >> 172.21.15.28:0/3129005090 conn(0x7fcd9c0f65e0 msgr2=0x7fcd9c0f8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:31:40.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:40.761+0000 7fcda32a7640 1 -- 172.21.15.28:0/3129005090 shutdown_connections 2024-06-18T02:31:40.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:40.761+0000 7fcda32a7640 1 -- 172.21.15.28:0/3129005090 wait complete. 2024-06-18T02:31:40.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:40.761+0000 7fcda32a7640 1 Processor -- start 2024-06-18T02:31:40.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:40.761+0000 7fcda32a7640 1 -- start start 2024-06-18T02:31:40.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:40.761+0000 7fcda32a7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd9c0fae70 0x7fcd9c065da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:31:40.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:40.761+0000 7fcda32a7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd9c10f2b0 con 0x7fcd9c0fae70 2024-06-18T02:31:40.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:40.761+0000 7fcda1043640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd9c0fae70 0x7fcd9c065da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:31:40.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:40.761+0000 7fcda1043640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd9c0fae70 0x7fcd9c065da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.28:44082/0 (socket says 172.21.15.28:44082) 2024-06-18T02:31:40.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:40.761+0000 7fcda1043640 1 -- 172.21.15.28:0/4058983942 learned_addr learned my addr 172.21.15.28:0/4058983942 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-06-18T02:31:40.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:40.762+0000 7fcda1043640 1 -- 172.21.15.28:0/4058983942 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd98005040 con 0x7fcd9c0fae70 2024-06-18T02:31:40.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:40.762+0000 7fcda1043640 1 --2- 172.21.15.28:0/4058983942 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd9c0fae70 0x7fcd9c065da0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fcd98004820 tx=0x7fcd98009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:31:40.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:40.762+0000 7fcd91ffb640 1 -- 172.21.15.28:0/4058983942 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd98032650 con 0x7fcd9c0fae70 2024-06-18T02:31:40.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:40.762+0000 7fcda32a7640 1 -- 172.21.15.28:0/4058983942 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd9c10f3f0 con 0x7fcd9c0fae70 2024-06-18T02:31:40.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:40.762+0000 7fcd91ffb640 1 -- 172.21.15.28:0/4058983942 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcd98032c80 con 0x7fcd9c0fae70 2024-06-18T02:31:40.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:40.762+0000 7fcd91ffb640 1 -- 172.21.15.28:0/4058983942 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd98033d60 con 0x7fcd9c0fae70 2024-06-18T02:31:40.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:40.762+0000 7fcda32a7640 1 -- 172.21.15.28:0/4058983942 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd9c10f860 con 0x7fcd9c0fae70 2024-06-18T02:31:40.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:40.763+0000 7fcd91ffb640 1 -- 172.21.15.28:0/4058983942 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) v1 ==== 42001+0+0 (secure 0 0 0) 0x7fcd98030020 con 0x7fcd9c0fae70 2024-06-18T02:31:40.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:40.763+0000 7fcd91ffb640 1 --2- 172.21.15.28:0/4058983942 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7fcd88034900 0x7fcd88036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:31:40.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:40.763+0000 7fcd91ffb640 1 -- 172.21.15.28:0/4058983942 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fcd980667e0 con 0x7fcd9c0fae70 2024-06-18T02:31:40.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:40.763+0000 7fcda32a7640 1 -- 172.21.15.28:0/4058983942 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd9c05ead0 con 0x7fcd9c0fae70 2024-06-18T02:31:40.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:40.764+0000 7fcda0842640 1 --2- 172.21.15.28:0/4058983942 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7fcd88034900 0x7fcd88036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:31:40.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:40.764+0000 7fcda0842640 1 --2- 172.21.15.28:0/4058983942 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7fcd88034900 0x7fcd88036db0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fcd8c006fd0 tx=0x7fcd8c006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:31:40.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:40.769+0000 7fcd91ffb640 1 -- 172.21.15.28:0/4058983942 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fcd9803c380 con 0x7fcd9c0fae70 2024-06-18T02:31:41.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:41.013+0000 7fcda32a7640 1 -- 172.21.15.28:0/4058983942 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcd9c100cc0 con 0x7fcd9c0fae70 2024-06-18T02:31:41.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:41.013+0000 7fcd91ffb640 1 -- 172.21.15.28:0/4058983942 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+735 (secure 0 0 0) 0x7fcd9802c110 con 0x7fcd9c0fae70 2024-06-18T02:31:41.014 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:31:41.014 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":1,"fsid":"7eefb63e-2d1a-11ef-bc9d-c7b262605968","modified":"2024-06-18T02:29:55.463546Z","created":"2024-06-18T02:29:55.463546Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi001","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-18T02:31:41.015 INFO:teuthology.orchestra.run.smithi028.stderr:dumped monmap epoch 1 2024-06-18T02:31:41.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:41.015+0000 7fcda32a7640 1 -- 172.21.15.28:0/4058983942 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7fcd88034900 msgr2=0x7fcd88036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:31:41.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:41.015+0000 7fcda32a7640 1 --2- 172.21.15.28:0/4058983942 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7fcd88034900 0x7fcd88036db0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fcd8c006fd0 tx=0x7fcd8c006db0 comp rx=0 tx=0).stop 2024-06-18T02:31:41.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:41.015+0000 7fcda32a7640 1 -- 172.21.15.28:0/4058983942 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd9c0fae70 msgr2=0x7fcd9c065da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:31:41.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:41.015+0000 7fcda32a7640 1 --2- 172.21.15.28:0/4058983942 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd9c0fae70 0x7fcd9c065da0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fcd98004820 tx=0x7fcd98009da0 comp rx=0 tx=0).stop 2024-06-18T02:31:41.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:41.015+0000 7fcda32a7640 1 -- 172.21.15.28:0/4058983942 shutdown_connections 2024-06-18T02:31:41.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:41.015+0000 7fcda32a7640 1 --2- 172.21.15.28:0/4058983942 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7fcd88034900 0x7fcd88036db0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:31:41.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:41.015+0000 7fcda32a7640 1 --2- 172.21.15.28:0/4058983942 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd9c0fae70 0x7fcd9c065da0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:31:41.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:41.015+0000 7fcda32a7640 1 -- 172.21.15.28:0/4058983942 >> 172.21.15.28:0/4058983942 conn(0x7fcd9c0f65e0 msgr2=0x7fcd9c101fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:31:41.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:41.015+0000 7fcda32a7640 1 -- 172.21.15.28:0/4058983942 shutdown_connections 2024-06-18T02:31:41.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:41.015+0000 7fcda32a7640 1 -- 172.21.15.28:0/4058983942 wait complete. 2024-06-18T02:31:41.377 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:41 smithi001 ceph-mon[29099]: from='client.? 172.21.15.28:0/4058983942' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-18T02:31:42.377 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:42 smithi001 ceph-mon[29099]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:31:42.491 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-18T02:31:42.492 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph mon dump -f json 2024-06-18T02:31:42.754 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-18T02:31:42.802 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-18T02:31:44.174 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:44 smithi001 ceph-mon[29099]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:31:44.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:44.196+0000 7f5a87a7d640 1 -- 172.21.15.28:0/3736276924 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a800fae60 msgr2=0x7f5a800fb270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:31:44.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:44.196+0000 7f5a87a7d640 1 --2- 172.21.15.28:0/3736276924 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a800fae60 0x7f5a800fb270 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f5a7c004850 tx=0x7f5a7c009da0 comp rx=0 tx=0).stop 2024-06-18T02:31:44.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:44.197+0000 7f5a87a7d640 1 -- 172.21.15.28:0/3736276924 shutdown_connections 2024-06-18T02:31:44.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:44.197+0000 7f5a87a7d640 1 --2- 172.21.15.28:0/3736276924 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a800fae60 0x7f5a800fb270 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:31:44.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:44.197+0000 7f5a87a7d640 1 -- 172.21.15.28:0/3736276924 >> 172.21.15.28:0/3736276924 conn(0x7f5a800f65b0 msgr2=0x7f5a800f8a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:31:44.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:44.197+0000 7f5a87a7d640 1 -- 172.21.15.28:0/3736276924 shutdown_connections 2024-06-18T02:31:44.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:44.197+0000 7f5a87a7d640 1 -- 172.21.15.28:0/3736276924 wait complete. 2024-06-18T02:31:44.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:44.197+0000 7f5a87a7d640 1 Processor -- start 2024-06-18T02:31:44.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:44.197+0000 7f5a87a7d640 1 -- start start 2024-06-18T02:31:44.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:44.198+0000 7f5a87a7d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a800fae60 0x7f5a8010c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:31:44.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:44.198+0000 7f5a87a7d640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a8010cad0 con 0x7f5a800fae60 2024-06-18T02:31:44.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:44.198+0000 7f5a85819640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a800fae60 0x7f5a8010c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:31:44.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:44.198+0000 7f5a85819640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a800fae60 0x7f5a8010c590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.28:37714/0 (socket says 172.21.15.28:37714) 2024-06-18T02:31:44.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:44.198+0000 7f5a85819640 1 -- 172.21.15.28:0/3182659969 learned_addr learned my addr 172.21.15.28:0/3182659969 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-06-18T02:31:44.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:44.198+0000 7f5a85819640 1 -- 172.21.15.28:0/3182659969 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a7c005040 con 0x7f5a800fae60 2024-06-18T02:31:44.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:44.198+0000 7f5a85819640 1 --2- 172.21.15.28:0/3182659969 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a800fae60 0x7f5a8010c590 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5a7c004820 tx=0x7f5a7c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:31:44.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:44.198+0000 7f5a767fc640 1 -- 172.21.15.28:0/3182659969 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a7c032650 con 0x7f5a800fae60 2024-06-18T02:31:44.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:44.198+0000 7f5a87a7d640 1 -- 172.21.15.28:0/3182659969 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a8010ccd0 con 0x7f5a800fae60 2024-06-18T02:31:44.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:44.198+0000 7f5a767fc640 1 -- 172.21.15.28:0/3182659969 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5a7c032c80 con 0x7f5a800fae60 2024-06-18T02:31:44.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:44.198+0000 7f5a767fc640 1 -- 172.21.15.28:0/3182659969 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a7c033d60 con 0x7f5a800fae60 2024-06-18T02:31:44.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:44.199+0000 7f5a87a7d640 1 -- 172.21.15.28:0/3182659969 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a8010d860 con 0x7f5a800fae60 2024-06-18T02:31:44.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:44.199+0000 7f5a767fc640 1 -- 172.21.15.28:0/3182659969 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) v1 ==== 42001+0+0 (secure 0 0 0) 0x7f5a7c030020 con 0x7f5a800fae60 2024-06-18T02:31:44.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:44.200+0000 7f5a767fc640 1 --2- 172.21.15.28:0/3182659969 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f5a6c034900 0x7f5a6c036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:31:44.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:44.200+0000 7f5a767fc640 1 -- 172.21.15.28:0/3182659969 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f5a7c0667f0 con 0x7f5a800fae60 2024-06-18T02:31:44.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:44.200+0000 7f5a87a7d640 1 -- 172.21.15.28:0/3182659969 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a80102490 con 0x7f5a800fae60 2024-06-18T02:31:44.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:44.200+0000 7f5a85018640 1 --2- 172.21.15.28:0/3182659969 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f5a6c034900 0x7f5a6c036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:31:44.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:44.201+0000 7f5a85018640 1 --2- 172.21.15.28:0/3182659969 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f5a6c034900 0x7f5a6c036db0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5a70006fd0 tx=0x7f5a70006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:31:44.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:44.205+0000 7f5a767fc640 1 -- 172.21.15.28:0/3182659969 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f5a7c03c380 con 0x7f5a800fae60 2024-06-18T02:31:44.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:44.442+0000 7f5a87a7d640 1 -- 172.21.15.28:0/3182659969 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5a8005ead0 con 0x7f5a800fae60 2024-06-18T02:31:44.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:44.442+0000 7f5a767fc640 1 -- 172.21.15.28:0/3182659969 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+735 (secure 0 0 0) 0x7f5a7c03b320 con 0x7f5a800fae60 2024-06-18T02:31:44.444 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:31:44.444 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":1,"fsid":"7eefb63e-2d1a-11ef-bc9d-c7b262605968","modified":"2024-06-18T02:29:55.463546Z","created":"2024-06-18T02:29:55.463546Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi001","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-18T02:31:44.445 INFO:teuthology.orchestra.run.smithi028.stderr:dumped monmap epoch 1 2024-06-18T02:31:44.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:44.444+0000 7f5a87a7d640 1 -- 172.21.15.28:0/3182659969 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f5a6c034900 msgr2=0x7f5a6c036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:31:44.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:44.444+0000 7f5a87a7d640 1 --2- 172.21.15.28:0/3182659969 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f5a6c034900 0x7f5a6c036db0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5a70006fd0 tx=0x7f5a70006db0 comp rx=0 tx=0).stop 2024-06-18T02:31:44.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:44.444+0000 7f5a87a7d640 1 -- 172.21.15.28:0/3182659969 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a800fae60 msgr2=0x7f5a8010c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:31:44.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:44.444+0000 7f5a87a7d640 1 --2- 172.21.15.28:0/3182659969 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a800fae60 0x7f5a8010c590 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5a7c004820 tx=0x7f5a7c009da0 comp rx=0 tx=0).stop 2024-06-18T02:31:44.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:44.444+0000 7f5a87a7d640 1 -- 172.21.15.28:0/3182659969 shutdown_connections 2024-06-18T02:31:44.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:44.444+0000 7f5a87a7d640 1 --2- 172.21.15.28:0/3182659969 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f5a6c034900 0x7f5a6c036db0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:31:44.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:44.444+0000 7f5a87a7d640 1 --2- 172.21.15.28:0/3182659969 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a800fae60 0x7f5a8010c590 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:31:44.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:44.444+0000 7f5a87a7d640 1 -- 172.21.15.28:0/3182659969 >> 172.21.15.28:0/3182659969 conn(0x7f5a800f65b0 msgr2=0x7f5a800f6e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:31:44.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:44.444+0000 7f5a87a7d640 1 -- 172.21.15.28:0/3182659969 shutdown_connections 2024-06-18T02:31:44.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:44.444+0000 7f5a87a7d640 1 -- 172.21.15.28:0/3182659969 wait complete. 2024-06-18T02:31:45.282 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:45 smithi001 ceph-mon[29099]: from='client.? 172.21.15.28:0/3182659969' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-18T02:31:46.109 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-18T02:31:46.109 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph mon dump -f json 2024-06-18T02:31:46.177 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:46 smithi001 ceph-mon[29099]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:31:46.177 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:46 smithi001 ceph-mon[29099]: from='mgr.14160 172.21.15.1:0/1498115122' entity='mgr.smithi001.teoawz' 2024-06-18T02:31:46.177 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:46 smithi001 ceph-mon[29099]: from='mgr.14160 172.21.15.1:0/1498115122' entity='mgr.smithi001.teoawz' 2024-06-18T02:31:46.177 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:46 smithi001 ceph-mon[29099]: from='mgr.14160 172.21.15.1:0/1498115122' entity='mgr.smithi001.teoawz' 2024-06-18T02:31:46.177 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:46 smithi001 ceph-mon[29099]: from='mgr.14160 172.21.15.1:0/1498115122' entity='mgr.smithi001.teoawz' 2024-06-18T02:31:46.178 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:46 smithi001 ceph-mon[29099]: from='mgr.14160 172.21.15.1:0/1498115122' entity='mgr.smithi001.teoawz' 2024-06-18T02:31:46.178 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:46 smithi001 ceph-mon[29099]: from='mgr.14160 172.21.15.1:0/1498115122' entity='mgr.smithi001.teoawz' 2024-06-18T02:31:46.372 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-18T02:31:46.420 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-18T02:31:47.377 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:47 smithi001 ceph-mon[29099]: Deploying daemon prometheus.smithi001 on smithi001 2024-06-18T02:31:47.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:47.665+0000 7f1d237fe640 1 -- 172.21.15.28:0/889295233 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d140326a0 con 0x7f1d240fb040 2024-06-18T02:31:47.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:47.665+0000 7f1d2a850640 1 -- 172.21.15.28:0/889295233 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d240fb040 msgr2=0x7f1d240fb450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:31:47.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:47.665+0000 7f1d2a850640 1 --2- 172.21.15.28:0/889295233 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d240fb040 0x7f1d240fb450 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f1d14004850 tx=0x7f1d14009da0 comp rx=0 tx=0).stop 2024-06-18T02:31:47.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:47.665+0000 7f1d2a850640 1 -- 172.21.15.28:0/889295233 shutdown_connections 2024-06-18T02:31:47.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:47.665+0000 7f1d2a850640 1 --2- 172.21.15.28:0/889295233 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d240fb040 0x7f1d240fb450 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:31:47.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:47.665+0000 7f1d2a850640 1 -- 172.21.15.28:0/889295233 >> 172.21.15.28:0/889295233 conn(0x7f1d240f6820 msgr2=0x7f1d240f8c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:31:47.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:47.666+0000 7f1d2a850640 1 -- 172.21.15.28:0/889295233 shutdown_connections 2024-06-18T02:31:47.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:47.666+0000 7f1d2a850640 1 -- 172.21.15.28:0/889295233 wait complete. 2024-06-18T02:31:47.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:47.666+0000 7f1d2a850640 1 Processor -- start 2024-06-18T02:31:47.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:47.666+0000 7f1d2a850640 1 -- start start 2024-06-18T02:31:47.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:47.666+0000 7f1d2a850640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d240fb040 0x7f1d24065290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:31:47.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:47.666+0000 7f1d2a850640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d2410f420 con 0x7f1d240fb040 2024-06-18T02:31:47.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:47.667+0000 7f1d23fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d240fb040 0x7f1d24065290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:31:47.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:47.667+0000 7f1d23fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d240fb040 0x7f1d24065290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.28:37728/0 (socket says 172.21.15.28:37728) 2024-06-18T02:31:47.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:47.667+0000 7f1d23fff640 1 -- 172.21.15.28:0/600104729 learned_addr learned my addr 172.21.15.28:0/600104729 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-06-18T02:31:47.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:47.667+0000 7f1d23fff640 1 -- 172.21.15.28:0/600104729 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d14005040 con 0x7f1d240fb040 2024-06-18T02:31:47.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:47.667+0000 7f1d23fff640 1 --2- 172.21.15.28:0/600104729 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d240fb040 0x7f1d24065290 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f1d14004820 tx=0x7f1d14009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:31:47.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:47.667+0000 7f1d217fa640 1 -- 172.21.15.28:0/600104729 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d14032650 con 0x7f1d240fb040 2024-06-18T02:31:47.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:47.667+0000 7f1d2a850640 1 -- 172.21.15.28:0/600104729 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d2410f560 con 0x7f1d240fb040 2024-06-18T02:31:47.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:47.667+0000 7f1d217fa640 1 -- 172.21.15.28:0/600104729 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1d14032c80 con 0x7f1d240fb040 2024-06-18T02:31:47.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:47.667+0000 7f1d217fa640 1 -- 172.21.15.28:0/600104729 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d1403eda0 con 0x7f1d240fb040 2024-06-18T02:31:47.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:47.667+0000 7f1d2a850640 1 -- 172.21.15.28:0/600104729 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d2410fa30 con 0x7f1d240fb040 2024-06-18T02:31:47.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:47.668+0000 7f1d217fa640 1 -- 172.21.15.28:0/600104729 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) v1 ==== 42001+0+0 (secure 0 0 0) 0x7f1d14030020 con 0x7f1d240fb040 2024-06-18T02:31:47.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:47.669+0000 7f1d217fa640 1 --2- 172.21.15.28:0/600104729 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f1d10034900 0x7f1d10036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:31:47.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:47.669+0000 7f1d2a850640 1 -- 172.21.15.28:0/600104729 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d2405ead0 con 0x7f1d240fb040 2024-06-18T02:31:47.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:47.669+0000 7f1d217fa640 1 -- 172.21.15.28:0/600104729 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f1d14066920 con 0x7f1d240fb040 2024-06-18T02:31:47.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:47.669+0000 7f1d1bfff640 1 --2- 172.21.15.28:0/600104729 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f1d10034900 0x7f1d10036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:31:47.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:47.669+0000 7f1d1bfff640 1 --2- 172.21.15.28:0/600104729 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f1d10034900 0x7f1d10036db0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f1d0c006fd0 tx=0x7f1d0c006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:31:47.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:47.676+0000 7f1d217fa640 1 -- 172.21.15.28:0/600104729 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f1d1403c330 con 0x7f1d240fb040 2024-06-18T02:31:47.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:47.914+0000 7f1d2a850640 1 -- 172.21.15.28:0/600104729 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1d2404b480 con 0x7f1d240fb040 2024-06-18T02:31:47.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:47.914+0000 7f1d217fa640 1 -- 172.21.15.28:0/600104729 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+735 (secure 0 0 0) 0x7f1d140319d0 con 0x7f1d240fb040 2024-06-18T02:31:47.914 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:31:47.914 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":1,"fsid":"7eefb63e-2d1a-11ef-bc9d-c7b262605968","modified":"2024-06-18T02:29:55.463546Z","created":"2024-06-18T02:29:55.463546Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi001","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-18T02:31:47.915 INFO:teuthology.orchestra.run.smithi028.stderr:dumped monmap epoch 1 2024-06-18T02:31:47.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:47.916+0000 7f1d2a850640 1 -- 172.21.15.28:0/600104729 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f1d10034900 msgr2=0x7f1d10036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:31:47.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:47.916+0000 7f1d2a850640 1 --2- 172.21.15.28:0/600104729 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f1d10034900 0x7f1d10036db0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f1d0c006fd0 tx=0x7f1d0c006db0 comp rx=0 tx=0).stop 2024-06-18T02:31:47.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:47.916+0000 7f1d2a850640 1 -- 172.21.15.28:0/600104729 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d240fb040 msgr2=0x7f1d24065290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:31:47.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:47.916+0000 7f1d2a850640 1 --2- 172.21.15.28:0/600104729 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d240fb040 0x7f1d24065290 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f1d14004820 tx=0x7f1d14009da0 comp rx=0 tx=0).stop 2024-06-18T02:31:47.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:47.916+0000 7f1d2a850640 1 -- 172.21.15.28:0/600104729 shutdown_connections 2024-06-18T02:31:47.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:47.916+0000 7f1d2a850640 1 --2- 172.21.15.28:0/600104729 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f1d10034900 0x7f1d10036db0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:31:47.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:47.916+0000 7f1d2a850640 1 --2- 172.21.15.28:0/600104729 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1d240fb040 0x7f1d24065290 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:31:47.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:47.916+0000 7f1d2a850640 1 -- 172.21.15.28:0/600104729 >> 172.21.15.28:0/600104729 conn(0x7f1d240f6820 msgr2=0x7f1d241021e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:31:47.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:47.916+0000 7f1d2a850640 1 -- 172.21.15.28:0/600104729 shutdown_connections 2024-06-18T02:31:47.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:47.917+0000 7f1d2a850640 1 -- 172.21.15.28:0/600104729 wait complete. 2024-06-18T02:31:48.279 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:48 smithi001 ceph-mon[29099]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:31:48.280 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:48 smithi001 ceph-mon[29099]: from='client.? 172.21.15.28:0/600104729' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-18T02:31:49.420 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-18T02:31:49.421 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph mon dump -f json 2024-06-18T02:31:49.691 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-18T02:31:49.749 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-18T02:31:50.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:50 smithi001 ceph-mon[29099]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:31:50.379 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:50 smithi001 ceph-mon[29099]: from='mgr.14160 172.21.15.1:0/1498115122' entity='mgr.smithi001.teoawz' 2024-06-18T02:31:51.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:51.116+0000 7f885f576640 1 -- 172.21.15.28:0/4157046824 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88580fbf10 msgr2=0x7f88580fc320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:31:51.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:51.116+0000 7f885f576640 1 --2- 172.21.15.28:0/4157046824 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88580fbf10 0x7f88580fc320 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f8854004850 tx=0x7f8854009da0 comp rx=0 tx=0).stop 2024-06-18T02:31:51.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:51.117+0000 7f885f576640 1 -- 172.21.15.28:0/4157046824 shutdown_connections 2024-06-18T02:31:51.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:51.117+0000 7f885f576640 1 --2- 172.21.15.28:0/4157046824 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88580fbf10 0x7f88580fc320 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:31:51.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:51.117+0000 7f885f576640 1 -- 172.21.15.28:0/4157046824 >> 172.21.15.28:0/4157046824 conn(0x7f88580f76f0 msgr2=0x7f88580f9b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:31:51.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:51.117+0000 7f885f576640 1 -- 172.21.15.28:0/4157046824 shutdown_connections 2024-06-18T02:31:51.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:51.117+0000 7f885f576640 1 -- 172.21.15.28:0/4157046824 wait complete. 2024-06-18T02:31:51.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:51.117+0000 7f885f576640 1 Processor -- start 2024-06-18T02:31:51.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:51.117+0000 7f885f576640 1 -- start start 2024-06-18T02:31:51.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:51.117+0000 7f885f576640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88580fbf10 0x7f885810c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:31:51.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:51.117+0000 7f885f576640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f885810ca50 con 0x7f88580fbf10 2024-06-18T02:31:51.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:51.118+0000 7f885d312640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88580fbf10 0x7f885810c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:31:51.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:51.118+0000 7f885d312640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88580fbf10 0x7f885810c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.28:37740/0 (socket says 172.21.15.28:37740) 2024-06-18T02:31:51.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:51.118+0000 7f885d312640 1 -- 172.21.15.28:0/2732257120 learned_addr learned my addr 172.21.15.28:0/2732257120 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-06-18T02:31:51.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:51.118+0000 7f885d312640 1 -- 172.21.15.28:0/2732257120 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8854005040 con 0x7f88580fbf10 2024-06-18T02:31:51.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:51.118+0000 7f885d312640 1 --2- 172.21.15.28:0/2732257120 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88580fbf10 0x7f885810c510 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f8854004820 tx=0x7f8854009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:31:51.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:51.118+0000 7f884dffb640 1 -- 172.21.15.28:0/2732257120 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8854032650 con 0x7f88580fbf10 2024-06-18T02:31:51.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:51.118+0000 7f885f576640 1 -- 172.21.15.28:0/2732257120 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f885810cc50 con 0x7f88580fbf10 2024-06-18T02:31:51.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:51.118+0000 7f884dffb640 1 -- 172.21.15.28:0/2732257120 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8854032c80 con 0x7f88580fbf10 2024-06-18T02:31:51.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:51.118+0000 7f884dffb640 1 -- 172.21.15.28:0/2732257120 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f885403eda0 con 0x7f88580fbf10 2024-06-18T02:31:51.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:51.118+0000 7f885f576640 1 -- 172.21.15.28:0/2732257120 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f885810d7e0 con 0x7f88580fbf10 2024-06-18T02:31:51.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:51.119+0000 7f884dffb640 1 -- 172.21.15.28:0/2732257120 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) v1 ==== 42001+0+0 (secure 0 0 0) 0x7f8854030020 con 0x7f88580fbf10 2024-06-18T02:31:51.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:51.119+0000 7f885f576640 1 -- 172.21.15.28:0/2732257120 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8858102480 con 0x7f88580fbf10 2024-06-18T02:31:51.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:51.120+0000 7f884dffb640 1 --2- 172.21.15.28:0/2732257120 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f8844034900 0x7f8844036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:31:51.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:51.120+0000 7f884dffb640 1 -- 172.21.15.28:0/2732257120 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f88540669e0 con 0x7f88580fbf10 2024-06-18T02:31:51.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:51.120+0000 7f885cb11640 1 --2- 172.21.15.28:0/2732257120 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f8844034900 0x7f8844036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:31:51.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:51.120+0000 7f885cb11640 1 --2- 172.21.15.28:0/2732257120 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f8844034900 0x7f8844036db0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8848006fd0 tx=0x7f8848006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:31:51.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:51.128+0000 7f884dffb640 1 -- 172.21.15.28:0/2732257120 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f88540327b0 con 0x7f88580fbf10 2024-06-18T02:31:51.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:51.370+0000 7f885f576640 1 -- 172.21.15.28:0/2732257120 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f885810e1e0 con 0x7f88580fbf10 2024-06-18T02:31:51.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:51.371+0000 7f884dffb640 1 -- 172.21.15.28:0/2732257120 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+735 (secure 0 0 0) 0x7f885403c330 con 0x7f88580fbf10 2024-06-18T02:31:51.371 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:31:51.371 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":1,"fsid":"7eefb63e-2d1a-11ef-bc9d-c7b262605968","modified":"2024-06-18T02:29:55.463546Z","created":"2024-06-18T02:29:55.463546Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi001","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-18T02:31:51.371 INFO:teuthology.orchestra.run.smithi028.stderr:dumped monmap epoch 1 2024-06-18T02:31:51.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:51.373+0000 7f885f576640 1 -- 172.21.15.28:0/2732257120 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f8844034900 msgr2=0x7f8844036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:31:51.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:51.373+0000 7f885f576640 1 --2- 172.21.15.28:0/2732257120 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f8844034900 0x7f8844036db0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8848006fd0 tx=0x7f8848006db0 comp rx=0 tx=0).stop 2024-06-18T02:31:51.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:51.373+0000 7f885f576640 1 -- 172.21.15.28:0/2732257120 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88580fbf10 msgr2=0x7f885810c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:31:51.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:51.373+0000 7f885f576640 1 --2- 172.21.15.28:0/2732257120 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88580fbf10 0x7f885810c510 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f8854004820 tx=0x7f8854009da0 comp rx=0 tx=0).stop 2024-06-18T02:31:51.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:51.373+0000 7f885f576640 1 -- 172.21.15.28:0/2732257120 shutdown_connections 2024-06-18T02:31:51.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:51.373+0000 7f885f576640 1 --2- 172.21.15.28:0/2732257120 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f8844034900 0x7f8844036db0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:31:51.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:51.373+0000 7f885f576640 1 --2- 172.21.15.28:0/2732257120 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88580fbf10 0x7f885810c510 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:31:51.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:51.373+0000 7f885f576640 1 -- 172.21.15.28:0/2732257120 >> 172.21.15.28:0/2732257120 conn(0x7f88580f76f0 msgr2=0x7f88581030b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:31:51.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:51.373+0000 7f885f576640 1 -- 172.21.15.28:0/2732257120 shutdown_connections 2024-06-18T02:31:51.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:51.373+0000 7f885f576640 1 -- 172.21.15.28:0/2732257120 wait complete. 2024-06-18T02:31:52.377 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:52 smithi001 ceph-mon[29099]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:31:52.377 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:52 smithi001 ceph-mon[29099]: from='client.? 172.21.15.28:0/2732257120' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-18T02:31:52.942 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-18T02:31:52.943 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph mon dump -f json 2024-06-18T02:31:53.205 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-18T02:31:53.254 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-18T02:31:54.377 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:54 smithi001 ceph-mon[29099]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:31:54.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:54.630+0000 7fb62634c640 1 -- 172.21.15.28:0/4106016964 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6200fc420 msgr2=0x7fb6200fc830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:31:54.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:54.630+0000 7fb62634c640 1 --2- 172.21.15.28:0/4106016964 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6200fc420 0x7fb6200fc830 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb610004850 tx=0x7fb610009da0 comp rx=0 tx=0).stop 2024-06-18T02:31:54.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:54.630+0000 7fb62634c640 1 -- 172.21.15.28:0/4106016964 shutdown_connections 2024-06-18T02:31:54.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:54.630+0000 7fb62634c640 1 --2- 172.21.15.28:0/4106016964 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6200fc420 0x7fb6200fc830 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:31:54.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:54.630+0000 7fb62634c640 1 -- 172.21.15.28:0/4106016964 >> 172.21.15.28:0/4106016964 conn(0x7fb6200f5da0 msgr2=0x7fb6200f81b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:31:54.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:54.630+0000 7fb62634c640 1 -- 172.21.15.28:0/4106016964 shutdown_connections 2024-06-18T02:31:54.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:54.630+0000 7fb62634c640 1 -- 172.21.15.28:0/4106016964 wait complete. 2024-06-18T02:31:54.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:54.630+0000 7fb62634c640 1 Processor -- start 2024-06-18T02:31:54.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:54.631+0000 7fb62634c640 1 -- start start 2024-06-18T02:31:54.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:54.631+0000 7fb62634c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6200fc420 0x7fb62010b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:31:54.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:54.631+0000 7fb62634c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb62010b990 con 0x7fb6200fc420 2024-06-18T02:31:54.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:54.631+0000 7fb61ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6200fc420 0x7fb62010b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:31:54.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:54.631+0000 7fb61ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6200fc420 0x7fb62010b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.28:49336/0 (socket says 172.21.15.28:49336) 2024-06-18T02:31:54.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:54.631+0000 7fb61ffff640 1 -- 172.21.15.28:0/1524356065 learned_addr learned my addr 172.21.15.28:0/1524356065 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-06-18T02:31:54.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:54.631+0000 7fb61ffff640 1 -- 172.21.15.28:0/1524356065 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb610005040 con 0x7fb6200fc420 2024-06-18T02:31:54.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:54.632+0000 7fb61ffff640 1 --2- 172.21.15.28:0/1524356065 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6200fc420 0x7fb62010b450 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fb610004820 tx=0x7fb610009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:31:54.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:54.632+0000 7fb61cff9640 1 -- 172.21.15.28:0/1524356065 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb610032650 con 0x7fb6200fc420 2024-06-18T02:31:54.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:54.632+0000 7fb61cff9640 1 -- 172.21.15.28:0/1524356065 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb610032c80 con 0x7fb6200fc420 2024-06-18T02:31:54.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:54.632+0000 7fb62634c640 1 -- 172.21.15.28:0/1524356065 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb62010bb90 con 0x7fb6200fc420 2024-06-18T02:31:54.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:54.632+0000 7fb61cff9640 1 -- 172.21.15.28:0/1524356065 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb61003eda0 con 0x7fb6200fc420 2024-06-18T02:31:54.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:54.632+0000 7fb62634c640 1 -- 172.21.15.28:0/1524356065 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb62010c720 con 0x7fb6200fc420 2024-06-18T02:31:54.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:54.633+0000 7fb61cff9640 1 -- 172.21.15.28:0/1524356065 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) v1 ==== 42001+0+0 (secure 0 0 0) 0x7fb610030020 con 0x7fb6200fc420 2024-06-18T02:31:54.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:54.633+0000 7fb61cff9640 1 --2- 172.21.15.28:0/1524356065 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7fb608034900 0x7fb608036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:31:54.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:54.633+0000 7fb62634c640 1 -- 172.21.15.28:0/1524356065 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb62010c060 con 0x7fb6200fc420 2024-06-18T02:31:54.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:54.633+0000 7fb61cff9640 1 -- 172.21.15.28:0/1524356065 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fb610066c20 con 0x7fb6200fc420 2024-06-18T02:31:54.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:54.633+0000 7fb61f7fe640 1 --2- 172.21.15.28:0/1524356065 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7fb608034900 0x7fb608036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:31:54.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:54.634+0000 7fb61f7fe640 1 --2- 172.21.15.28:0/1524356065 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7fb608034900 0x7fb608036db0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb614006fd0 tx=0x7fb614006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:31:54.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:54.638+0000 7fb61cff9640 1 -- 172.21.15.28:0/1524356065 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fb62010c060 con 0x7fb6200fc420 2024-06-18T02:31:54.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:54.882+0000 7fb62634c640 1 -- 172.21.15.28:0/1524356065 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb6201013d0 con 0x7fb6200fc420 2024-06-18T02:31:54.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:54.883+0000 7fb61cff9640 1 -- 172.21.15.28:0/1524356065 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+735 (secure 0 0 0) 0x7fb610033420 con 0x7fb6200fc420 2024-06-18T02:31:54.883 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:31:54.883 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":1,"fsid":"7eefb63e-2d1a-11ef-bc9d-c7b262605968","modified":"2024-06-18T02:29:55.463546Z","created":"2024-06-18T02:29:55.463546Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi001","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-18T02:31:54.883 INFO:teuthology.orchestra.run.smithi028.stderr:dumped monmap epoch 1 2024-06-18T02:31:54.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:54.885+0000 7fb62634c640 1 -- 172.21.15.28:0/1524356065 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7fb608034900 msgr2=0x7fb608036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:31:54.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:54.885+0000 7fb62634c640 1 --2- 172.21.15.28:0/1524356065 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7fb608034900 0x7fb608036db0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb614006fd0 tx=0x7fb614006db0 comp rx=0 tx=0).stop 2024-06-18T02:31:54.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:54.885+0000 7fb62634c640 1 -- 172.21.15.28:0/1524356065 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6200fc420 msgr2=0x7fb62010b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:31:54.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:54.885+0000 7fb62634c640 1 --2- 172.21.15.28:0/1524356065 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6200fc420 0x7fb62010b450 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fb610004820 tx=0x7fb610009da0 comp rx=0 tx=0).stop 2024-06-18T02:31:54.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:54.885+0000 7fb62634c640 1 -- 172.21.15.28:0/1524356065 shutdown_connections 2024-06-18T02:31:54.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:54.885+0000 7fb62634c640 1 --2- 172.21.15.28:0/1524356065 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7fb608034900 0x7fb608036db0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:31:54.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:54.885+0000 7fb62634c640 1 --2- 172.21.15.28:0/1524356065 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6200fc420 0x7fb62010b450 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:31:54.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:54.885+0000 7fb62634c640 1 -- 172.21.15.28:0/1524356065 >> 172.21.15.28:0/1524356065 conn(0x7fb6200f5da0 msgr2=0x7fb6200f6900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:31:54.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:54.885+0000 7fb62634c640 1 -- 172.21.15.28:0/1524356065 shutdown_connections 2024-06-18T02:31:54.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:54.885+0000 7fb62634c640 1 -- 172.21.15.28:0/1524356065 wait complete. 2024-06-18T02:31:55.127 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:55 smithi001 ceph-mon[29099]: from='client.? 172.21.15.28:0/1524356065' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-18T02:31:56.127 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:56 smithi001 ceph-mon[29099]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:31:56.512 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-18T02:31:56.513 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph mon dump -f json 2024-06-18T02:31:56.774 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-18T02:31:56.822 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-18T02:31:58.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:58.241+0000 7f8c395f6640 1 -- 172.21.15.28:0/11973507 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c340fbf30 msgr2=0x7f8c340fc340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:31:58.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:58.241+0000 7f8c395f6640 1 --2- 172.21.15.28:0/11973507 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c340fbf30 0x7f8c340fc340 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f8c24004850 tx=0x7f8c24009da0 comp rx=0 tx=0).stop 2024-06-18T02:31:58.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:58.241+0000 7f8c395f6640 1 -- 172.21.15.28:0/11973507 shutdown_connections 2024-06-18T02:31:58.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:58.241+0000 7f8c395f6640 1 --2- 172.21.15.28:0/11973507 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c340fbf30 0x7f8c340fc340 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:31:58.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:58.241+0000 7f8c395f6640 1 -- 172.21.15.28:0/11973507 >> 172.21.15.28:0/11973507 conn(0x7f8c340f7710 msgr2=0x7f8c340f9b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:31:58.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:58.241+0000 7f8c395f6640 1 -- 172.21.15.28:0/11973507 shutdown_connections 2024-06-18T02:31:58.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:58.241+0000 7f8c395f6640 1 -- 172.21.15.28:0/11973507 wait complete. 2024-06-18T02:31:58.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:58.241+0000 7f8c395f6640 1 Processor -- start 2024-06-18T02:31:58.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:58.241+0000 7f8c395f6640 1 -- start start 2024-06-18T02:31:58.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:58.242+0000 7f8c395f6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c340fbf30 0x7f8c3410c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:31:58.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:58.242+0000 7f8c395f6640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c3410ca30 con 0x7f8c340fbf30 2024-06-18T02:31:58.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:58.242+0000 7f8c32ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c340fbf30 0x7f8c3410c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:31:58.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:58.242+0000 7f8c32ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c340fbf30 0x7f8c3410c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.28:49348/0 (socket says 172.21.15.28:49348) 2024-06-18T02:31:58.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:58.242+0000 7f8c32ffd640 1 -- 172.21.15.28:0/3847582668 learned_addr learned my addr 172.21.15.28:0/3847582668 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-06-18T02:31:58.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:58.242+0000 7f8c32ffd640 1 -- 172.21.15.28:0/3847582668 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c24005040 con 0x7f8c340fbf30 2024-06-18T02:31:58.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:58.242+0000 7f8c32ffd640 1 --2- 172.21.15.28:0/3847582668 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c340fbf30 0x7f8c3410c4f0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8c240098d0 tx=0x7f8c24009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:31:58.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:58.243+0000 7f8c1b7fe640 1 -- 172.21.15.28:0/3847582668 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8c24032650 con 0x7f8c340fbf30 2024-06-18T02:31:58.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:58.243+0000 7f8c1b7fe640 1 -- 172.21.15.28:0/3847582668 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8c24032c80 con 0x7f8c340fbf30 2024-06-18T02:31:58.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:58.243+0000 7f8c395f6640 1 -- 172.21.15.28:0/3847582668 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c3410cc30 con 0x7f8c340fbf30 2024-06-18T02:31:58.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:58.243+0000 7f8c1b7fe640 1 -- 172.21.15.28:0/3847582668 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8c24033c10 con 0x7f8c340fbf30 2024-06-18T02:31:58.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:58.243+0000 7f8c395f6640 1 -- 172.21.15.28:0/3847582668 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c3410d0a0 con 0x7f8c340fbf30 2024-06-18T02:31:58.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:58.243+0000 7f8c1b7fe640 1 -- 172.21.15.28:0/3847582668 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 13) v1 ==== 42001+0+0 (secure 0 0 0) 0x7f8c24030020 con 0x7f8c340fbf30 2024-06-18T02:31:58.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:58.244+0000 7f8c395f6640 1 -- 172.21.15.28:0/3847582668 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c3405ead0 con 0x7f8c340fbf30 2024-06-18T02:31:58.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:58.244+0000 7f8c1b7fe640 1 --2- 172.21.15.28:0/3847582668 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f8c1c034950 0x7f8c1c036e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:31:58.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:58.244+0000 7f8c1b7fe640 1 -- 172.21.15.28:0/3847582668 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f8c240666c0 con 0x7f8c340fbf30 2024-06-18T02:31:58.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:58.244+0000 7f8c327fc640 1 --2- 172.21.15.28:0/3847582668 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f8c1c034950 0x7f8c1c036e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:31:58.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:58.245+0000 7f8c327fc640 1 --2- 172.21.15.28:0/3847582668 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f8c1c034950 0x7f8c1c036e00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f8c28006fd0 tx=0x7f8c28006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:31:58.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:58.250+0000 7f8c1b7fe640 1 -- 172.21.15.28:0/3847582668 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f8c240304b0 con 0x7f8c340fbf30 2024-06-18T02:31:58.377 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:58 smithi001 ceph-mon[29099]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:31:58.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:58.491+0000 7f8c395f6640 1 -- 172.21.15.28:0/3847582668 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8c3404b480 con 0x7f8c340fbf30 2024-06-18T02:31:58.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:58.492+0000 7f8c1b7fe640 1 -- 172.21.15.28:0/3847582668 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+735 (secure 0 0 0) 0x7f8c240302d0 con 0x7f8c340fbf30 2024-06-18T02:31:58.493 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:31:58.493 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":1,"fsid":"7eefb63e-2d1a-11ef-bc9d-c7b262605968","modified":"2024-06-18T02:29:55.463546Z","created":"2024-06-18T02:29:55.463546Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi001","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-18T02:31:58.493 INFO:teuthology.orchestra.run.smithi028.stderr:dumped monmap epoch 1 2024-06-18T02:31:58.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:58.495+0000 7f8c395f6640 1 -- 172.21.15.28:0/3847582668 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f8c1c034950 msgr2=0x7f8c1c036e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:31:58.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:58.495+0000 7f8c395f6640 1 --2- 172.21.15.28:0/3847582668 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f8c1c034950 0x7f8c1c036e00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f8c28006fd0 tx=0x7f8c28006db0 comp rx=0 tx=0).stop 2024-06-18T02:31:58.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:58.495+0000 7f8c395f6640 1 -- 172.21.15.28:0/3847582668 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c340fbf30 msgr2=0x7f8c3410c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:31:58.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:58.495+0000 7f8c395f6640 1 --2- 172.21.15.28:0/3847582668 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c340fbf30 0x7f8c3410c4f0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8c240098d0 tx=0x7f8c24009da0 comp rx=0 tx=0).stop 2024-06-18T02:31:58.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:58.495+0000 7f8c395f6640 1 -- 172.21.15.28:0/3847582668 shutdown_connections 2024-06-18T02:31:58.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:58.495+0000 7f8c395f6640 1 --2- 172.21.15.28:0/3847582668 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f8c1c034950 0x7f8c1c036e00 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:31:58.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:58.495+0000 7f8c395f6640 1 --2- 172.21.15.28:0/3847582668 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c340fbf30 0x7f8c3410c4f0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:31:58.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:58.495+0000 7f8c395f6640 1 -- 172.21.15.28:0/3847582668 >> 172.21.15.28:0/3847582668 conn(0x7f8c340f7710 msgr2=0x7f8c341030d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:31:58.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:58.495+0000 7f8c395f6640 1 -- 172.21.15.28:0/3847582668 shutdown_connections 2024-06-18T02:31:58.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:31:58.495+0000 7f8c395f6640 1 -- 172.21.15.28:0/3847582668 wait complete. 2024-06-18T02:31:59.377 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:31:59 smithi001 ceph-mon[29099]: from='client.? 172.21.15.28:0/3847582668' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-18T02:32:00.092 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:00 smithi001 ceph-mon[29099]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:32:00.108 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-18T02:32:00.109 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph mon dump -f json 2024-06-18T02:32:00.372 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-18T02:32:00.421 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-18T02:32:01.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:01.870+0000 7fda1ad0f640 1 -- 172.21.15.28:0/3513821336 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda0c094870 msgr2=0x7fda0c094c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:32:01.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:01.870+0000 7fda1ad0f640 1 --2- 172.21.15.28:0/3513821336 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda0c094870 0x7fda0c094c80 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fda10004850 tx=0x7fda10009da0 comp rx=0 tx=0).stop 2024-06-18T02:32:01.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:01.870+0000 7fda1ad0f640 1 -- 172.21.15.28:0/3513821336 shutdown_connections 2024-06-18T02:32:01.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:01.870+0000 7fda1ad0f640 1 --2- 172.21.15.28:0/3513821336 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda0c094870 0x7fda0c094c80 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:32:01.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:01.870+0000 7fda1ad0f640 1 -- 172.21.15.28:0/3513821336 >> 172.21.15.28:0/3513821336 conn(0x7fda0c08ff90 msgr2=0x7fda0c0923c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:32:01.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:01.871+0000 7fda1ad0f640 1 -- 172.21.15.28:0/3513821336 shutdown_connections 2024-06-18T02:32:01.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:01.871+0000 7fda1ad0f640 1 -- 172.21.15.28:0/3513821336 wait complete. 2024-06-18T02:32:01.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:01.871+0000 7fda1ad0f640 1 Processor -- start 2024-06-18T02:32:01.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:01.871+0000 7fda1ad0f640 1 -- start start 2024-06-18T02:32:01.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:01.872+0000 7fda1ad0f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda0c094870 0x7fda0c0a6fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:32:01.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:01.872+0000 7fda1ad0f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda0c0a74f0 con 0x7fda0c094870 2024-06-18T02:32:01.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:01.872+0000 7fda19d0d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda0c094870 0x7fda0c0a6fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:32:01.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:01.872+0000 7fda19d0d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda0c094870 0x7fda0c0a6fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.28:49358/0 (socket says 172.21.15.28:49358) 2024-06-18T02:32:01.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:01.872+0000 7fda19d0d640 1 -- 172.21.15.28:0/894173194 learned_addr learned my addr 172.21.15.28:0/894173194 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-06-18T02:32:01.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:01.872+0000 7fda19d0d640 1 -- 172.21.15.28:0/894173194 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda10005040 con 0x7fda0c094870 2024-06-18T02:32:01.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:01.872+0000 7fda19d0d640 1 --2- 172.21.15.28:0/894173194 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda0c094870 0x7fda0c0a6fb0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fda1002cff0 tx=0x7fda10009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:32:01.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:01.873+0000 7fda0a7fc640 1 -- 172.21.15.28:0/894173194 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda10032650 con 0x7fda0c094870 2024-06-18T02:32:01.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:01.873+0000 7fda0a7fc640 1 -- 172.21.15.28:0/894173194 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fda10032c80 con 0x7fda0c094870 2024-06-18T02:32:01.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:01.873+0000 7fda1ad0f640 1 -- 172.21.15.28:0/894173194 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda0c0a76f0 con 0x7fda0c094870 2024-06-18T02:32:01.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:01.873+0000 7fda0a7fc640 1 -- 172.21.15.28:0/894173194 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda1003dc40 con 0x7fda0c094870 2024-06-18T02:32:01.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:01.873+0000 7fda1ad0f640 1 -- 172.21.15.28:0/894173194 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda0c0a7bc0 con 0x7fda0c094870 2024-06-18T02:32:01.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:01.874+0000 7fda0a7fc640 1 -- 172.21.15.28:0/894173194 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 14) v1 ==== 42015+0+0 (secure 0 0 0) 0x7fda10030020 con 0x7fda0c094870 2024-06-18T02:32:01.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:01.874+0000 7fda1ad0f640 1 -- 172.21.15.28:0/894173194 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda0c004c10 con 0x7fda0c094870 2024-06-18T02:32:01.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:01.874+0000 7fda0a7fc640 1 --2- 172.21.15.28:0/894173194 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7fda00034a00 0x7fda00036eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:32:01.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:01.874+0000 7fda0a7fc640 1 -- 172.21.15.28:0/894173194 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fda10066830 con 0x7fda0c094870 2024-06-18T02:32:01.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:01.874+0000 7fda1950c640 1 -- 172.21.15.28:0/894173194 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7fda00034a00 msgr2=0x7fda00036eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/85550937 2024-06-18T02:32:01.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:01.874+0000 7fda1950c640 1 --2- 172.21.15.28:0/894173194 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7fda00034a00 0x7fda00036eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-18T02:32:01.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:01.879+0000 7fda0a7fc640 1 -- 172.21.15.28:0/894173194 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fda1003dda0 con 0x7fda0c094870 2024-06-18T02:32:02.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:02.075+0000 7fda1950c640 1 -- 172.21.15.28:0/894173194 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7fda00034a00 msgr2=0x7fda00036eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/85550937 2024-06-18T02:32:02.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:02.075+0000 7fda1950c640 1 --2- 172.21.15.28:0/894173194 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7fda00034a00 0x7fda00036eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-18T02:32:02.113 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:01 smithi001 ceph-mon[29099]: from='mgr.14160 172.21.15.1:0/1498115122' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:02.113 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:01 smithi001 ceph-mon[29099]: from='mgr.14160 172.21.15.1:0/1498115122' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:02.113 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:01 smithi001 ceph-mon[29099]: from='mgr.14160 172.21.15.1:0/1498115122' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:02.113 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:01 smithi001 ceph-mon[29099]: from='mgr.14160 172.21.15.1:0/1498115122' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-06-18T02:32:02.113 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:01 smithi001 ceph-mon[29099]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:32:02.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:02.119+0000 7fda1ad0f640 1 -- 172.21.15.28:0/894173194 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fda0c09a6a0 con 0x7fda0c094870 2024-06-18T02:32:02.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:02.119+0000 7fda0a7fc640 1 -- 172.21.15.28:0/894173194 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+735 (secure 0 0 0) 0x7fda10045af0 con 0x7fda0c094870 2024-06-18T02:32:02.119 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:32:02.119 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":1,"fsid":"7eefb63e-2d1a-11ef-bc9d-c7b262605968","modified":"2024-06-18T02:29:55.463546Z","created":"2024-06-18T02:29:55.463546Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi001","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-18T02:32:02.119 INFO:teuthology.orchestra.run.smithi028.stderr:dumped monmap epoch 1 2024-06-18T02:32:02.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:02.121+0000 7fda1ad0f640 1 -- 172.21.15.28:0/894173194 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7fda00034a00 msgr2=0x7fda00036eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-18T02:32:02.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:02.121+0000 7fda1ad0f640 1 --2- 172.21.15.28:0/894173194 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7fda00034a00 0x7fda00036eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:32:02.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:02.121+0000 7fda1ad0f640 1 -- 172.21.15.28:0/894173194 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda0c094870 msgr2=0x7fda0c0a6fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:32:02.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:02.121+0000 7fda1ad0f640 1 --2- 172.21.15.28:0/894173194 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda0c094870 0x7fda0c0a6fb0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fda1002cff0 tx=0x7fda10009da0 comp rx=0 tx=0).stop 2024-06-18T02:32:02.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:02.121+0000 7fda1ad0f640 1 -- 172.21.15.28:0/894173194 shutdown_connections 2024-06-18T02:32:02.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:02.121+0000 7fda1ad0f640 1 --2- 172.21.15.28:0/894173194 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7fda00034a00 0x7fda00036eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:32:02.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:02.121+0000 7fda1ad0f640 1 --2- 172.21.15.28:0/894173194 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda0c094870 0x7fda0c0a6fb0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:32:02.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:02.121+0000 7fda1ad0f640 1 -- 172.21.15.28:0/894173194 >> 172.21.15.28:0/894173194 conn(0x7fda0c08ff90 msgr2=0x7fda0c092370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:32:02.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:02.121+0000 7fda1ad0f640 1 -- 172.21.15.28:0/894173194 shutdown_connections 2024-06-18T02:32:02.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:02.121+0000 7fda1ad0f640 1 -- 172.21.15.28:0/894173194 wait complete. 2024-06-18T02:32:03.127 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:02 smithi001 ceph-mon[29099]: from='mgr.14160 172.21.15.1:0/1498115122' entity='mgr.smithi001.teoawz' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-06-18T02:32:03.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:02 smithi001 ceph-mon[29099]: mgrmap e14: smithi001.teoawz(active, since 62s) 2024-06-18T02:32:03.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:02 smithi001 ceph-mon[29099]: from='client.? 172.21.15.28:0/894173194' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-18T02:32:03.666 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-18T02:32:03.666 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph mon dump -f json 2024-06-18T02:32:03.928 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-18T02:32:03.977 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-18T02:32:05.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:05.266+0000 7f03455f3640 1 -- 172.21.15.28:0/836281742 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03400fbf10 msgr2=0x7f03400fc320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:32:05.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:05.266+0000 7f03455f3640 1 --2- 172.21.15.28:0/836281742 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03400fbf10 0x7f03400fc320 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f0338004850 tx=0x7f0338009da0 comp rx=0 tx=0).stop 2024-06-18T02:32:05.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:05.267+0000 7f03455f3640 1 -- 172.21.15.28:0/836281742 shutdown_connections 2024-06-18T02:32:05.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:05.267+0000 7f03455f3640 1 --2- 172.21.15.28:0/836281742 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03400fbf10 0x7f03400fc320 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:32:05.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:05.267+0000 7f03455f3640 1 -- 172.21.15.28:0/836281742 >> 172.21.15.28:0/836281742 conn(0x7f03400f76f0 msgr2=0x7f03400f9b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:32:05.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:05.267+0000 7f03455f3640 1 -- 172.21.15.28:0/836281742 shutdown_connections 2024-06-18T02:32:05.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:05.267+0000 7f03455f3640 1 -- 172.21.15.28:0/836281742 wait complete. 2024-06-18T02:32:05.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:05.267+0000 7f03455f3640 1 Processor -- start 2024-06-18T02:32:05.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:05.267+0000 7f03455f3640 1 -- start start 2024-06-18T02:32:05.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:05.267+0000 7f03455f3640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03400fbf10 0x7f034010e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:32:05.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:05.267+0000 7f03455f3640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f034010ec50 con 0x7f03400fbf10 2024-06-18T02:32:05.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:05.268+0000 7f033effd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03400fbf10 0x7f034010e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:32:05.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:05.268+0000 7f033effd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03400fbf10 0x7f034010e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.28:35534/0 (socket says 172.21.15.28:35534) 2024-06-18T02:32:05.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:05.268+0000 7f033effd640 1 -- 172.21.15.28:0/664186095 learned_addr learned my addr 172.21.15.28:0/664186095 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-06-18T02:32:05.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:05.268+0000 7f033effd640 1 -- 172.21.15.28:0/664186095 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0338005040 con 0x7f03400fbf10 2024-06-18T02:32:05.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:05.268+0000 7f033effd640 1 --2- 172.21.15.28:0/664186095 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03400fbf10 0x7f034010e710 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f0338004820 tx=0x7f0338009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:32:05.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:05.268+0000 7f03377fe640 1 -- 172.21.15.28:0/664186095 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0338032650 con 0x7f03400fbf10 2024-06-18T02:32:05.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:05.268+0000 7f03377fe640 1 -- 172.21.15.28:0/664186095 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0338032c80 con 0x7f03400fbf10 2024-06-18T02:32:05.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:05.268+0000 7f03455f3640 1 -- 172.21.15.28:0/664186095 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f034010ee50 con 0x7f03400fbf10 2024-06-18T02:32:05.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:05.268+0000 7f03377fe640 1 -- 172.21.15.28:0/664186095 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f033803eda0 con 0x7f03400fbf10 2024-06-18T02:32:05.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:05.268+0000 7f03455f3640 1 -- 172.21.15.28:0/664186095 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f034010f9e0 con 0x7f03400fbf10 2024-06-18T02:32:05.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:05.269+0000 7f03377fe640 1 -- 172.21.15.28:0/664186095 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 14) v1 ==== 42015+0+0 (secure 0 0 0) 0x7f0338030020 con 0x7f03400fbf10 2024-06-18T02:32:05.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:05.269+0000 7f03455f3640 1 -- 172.21.15.28:0/664186095 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0340102480 con 0x7f03400fbf10 2024-06-18T02:32:05.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:05.269+0000 7f03377fe640 1 --2- 172.21.15.28:0/664186095 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f032c0349a0 0x7f032c036e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:32:05.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:05.269+0000 7f03377fe640 1 -- 172.21.15.28:0/664186095 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f0338066a30 con 0x7f03400fbf10 2024-06-18T02:32:05.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:05.269+0000 7f033e7fc640 1 -- 172.21.15.28:0/664186095 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f032c0349a0 msgr2=0x7f032c036e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/85550937 2024-06-18T02:32:05.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:05.269+0000 7f033e7fc640 1 --2- 172.21.15.28:0/664186095 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f032c0349a0 0x7f032c036e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-18T02:32:05.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:05.274+0000 7f03377fe640 1 -- 172.21.15.28:0/664186095 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f033803e430 con 0x7f03400fbf10 2024-06-18T02:32:05.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:05.470+0000 7f033e7fc640 1 -- 172.21.15.28:0/664186095 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f032c0349a0 msgr2=0x7f032c036e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:6800/85550937 2024-06-18T02:32:05.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:05.470+0000 7f033e7fc640 1 --2- 172.21.15.28:0/664186095 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f032c0349a0 0x7f032c036e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-18T02:32:05.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:05.516+0000 7f03455f3640 1 -- 172.21.15.28:0/664186095 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f034010fc20 con 0x7f03400fbf10 2024-06-18T02:32:05.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:05.517+0000 7f03377fe640 1 -- 172.21.15.28:0/664186095 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+735 (secure 0 0 0) 0x7f033803c330 con 0x7f03400fbf10 2024-06-18T02:32:05.517 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:32:05.517 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":1,"fsid":"7eefb63e-2d1a-11ef-bc9d-c7b262605968","modified":"2024-06-18T02:29:55.463546Z","created":"2024-06-18T02:29:55.463546Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi001","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-18T02:32:05.518 INFO:teuthology.orchestra.run.smithi028.stderr:dumped monmap epoch 1 2024-06-18T02:32:05.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:05.519+0000 7f03455f3640 1 -- 172.21.15.28:0/664186095 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f032c0349a0 msgr2=0x7f032c036e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-18T02:32:05.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:05.519+0000 7f03455f3640 1 --2- 172.21.15.28:0/664186095 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f032c0349a0 0x7f032c036e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:32:05.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:05.519+0000 7f03455f3640 1 -- 172.21.15.28:0/664186095 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03400fbf10 msgr2=0x7f034010e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:32:05.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:05.519+0000 7f03455f3640 1 --2- 172.21.15.28:0/664186095 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03400fbf10 0x7f034010e710 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f0338004820 tx=0x7f0338009da0 comp rx=0 tx=0).stop 2024-06-18T02:32:05.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:05.519+0000 7f03455f3640 1 -- 172.21.15.28:0/664186095 shutdown_connections 2024-06-18T02:32:05.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:05.519+0000 7f03455f3640 1 --2- 172.21.15.28:0/664186095 >> [v2:172.21.15.1:6800/85550937,v1:172.21.15.1:6801/85550937] conn(0x7f032c0349a0 0x7f032c036e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:32:05.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:05.519+0000 7f03455f3640 1 --2- 172.21.15.28:0/664186095 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03400fbf10 0x7f034010e710 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:32:05.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:05.519+0000 7f03455f3640 1 -- 172.21.15.28:0/664186095 >> 172.21.15.28:0/664186095 conn(0x7f03400f76f0 msgr2=0x7f03400f9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:32:05.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:05.519+0000 7f03455f3640 1 -- 172.21.15.28:0/664186095 shutdown_connections 2024-06-18T02:32:05.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:05.519+0000 7f03455f3640 1 -- 172.21.15.28:0/664186095 wait complete. 2024-06-18T02:32:05.579 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:05 smithi001 ceph-mon[29099]: from='client.? 172.21.15.28:0/664186095' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-18T02:32:06.627 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:06 smithi001 ceph-mon[29099]: Active manager daemon smithi001.teoawz restarted 2024-06-18T02:32:06.628 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:06 smithi001 ceph-mon[29099]: Activating manager daemon smithi001.teoawz 2024-06-18T02:32:06.628 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:06 smithi001 ceph-mon[29099]: osdmap e5: 0 total, 0 up, 0 in 2024-06-18T02:32:06.628 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:06 smithi001 ceph-mon[29099]: mgrmap e15: smithi001.teoawz(active, starting, since 0.0029063s) 2024-06-18T02:32:06.628 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:06 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "mon metadata", "id": "smithi001"}]: dispatch 2024-06-18T02:32:06.628 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:06 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "mgr metadata", "who": "smithi001.teoawz", "id": "smithi001.teoawz"}]: dispatch 2024-06-18T02:32:06.628 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:06 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-18T02:32:06.628 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:06 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-18T02:32:06.629 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:06 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-18T02:32:06.629 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:06 smithi001 ceph-mon[29099]: Manager daemon smithi001.teoawz is now available 2024-06-18T02:32:06.629 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:06 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:32:06.629 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:06 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:32:06.629 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:06 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:06.629 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:06 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:06.629 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:06 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:32:06.629 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:06 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi001.teoawz/mirror_snapshot_schedule"}]: dispatch 2024-06-18T02:32:06.629 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:06 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi001.teoawz/trash_purge_schedule"}]: dispatch 2024-06-18T02:32:07.201 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-18T02:32:07.201 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph mon dump -f json 2024-06-18T02:32:07.456 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-18T02:32:07.507 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-18T02:32:08.424 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:08 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:08.425 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:08 smithi001 ceph-mon[29099]: mgrmap e16: smithi001.teoawz(active, since 1.00586s) 2024-06-18T02:32:08.425 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:08 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:09.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:09.029+0000 7fa461e8a640 1 -- 172.21.15.28:0/1581294978 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa45c0fbf30 msgr2=0x7fa45c0fc340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:32:09.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:09.029+0000 7fa461e8a640 1 --2- 172.21.15.28:0/1581294978 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa45c0fbf30 0x7fa45c0fc340 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa44c004850 tx=0x7fa44c009da0 comp rx=0 tx=0).stop 2024-06-18T02:32:09.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:09.030+0000 7fa461e8a640 1 -- 172.21.15.28:0/1581294978 shutdown_connections 2024-06-18T02:32:09.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:09.030+0000 7fa461e8a640 1 --2- 172.21.15.28:0/1581294978 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa45c0fbf30 0x7fa45c0fc340 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:32:09.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:09.030+0000 7fa461e8a640 1 -- 172.21.15.28:0/1581294978 >> 172.21.15.28:0/1581294978 conn(0x7fa45c0f7710 msgr2=0x7fa45c0f9b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:32:09.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:09.030+0000 7fa461e8a640 1 -- 172.21.15.28:0/1581294978 shutdown_connections 2024-06-18T02:32:09.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:09.030+0000 7fa461e8a640 1 -- 172.21.15.28:0/1581294978 wait complete. 2024-06-18T02:32:09.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:09.030+0000 7fa461e8a640 1 Processor -- start 2024-06-18T02:32:09.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:09.030+0000 7fa461e8a640 1 -- start start 2024-06-18T02:32:09.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:09.030+0000 7fa461e8a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa45c0fbf30 0x7fa45c10c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:32:09.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:09.030+0000 7fa461e8a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa45c10ca30 con 0x7fa45c0fbf30 2024-06-18T02:32:09.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:09.031+0000 7fa45b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa45c0fbf30 0x7fa45c10c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:32:09.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:09.031+0000 7fa45b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa45c0fbf30 0x7fa45c10c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.28:35558/0 (socket says 172.21.15.28:35558) 2024-06-18T02:32:09.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:09.031+0000 7fa45b7fe640 1 -- 172.21.15.28:0/2202071474 learned_addr learned my addr 172.21.15.28:0/2202071474 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-06-18T02:32:09.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:09.031+0000 7fa45b7fe640 1 -- 172.21.15.28:0/2202071474 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa44c005040 con 0x7fa45c0fbf30 2024-06-18T02:32:09.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:09.031+0000 7fa45b7fe640 1 --2- 172.21.15.28:0/2202071474 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa45c0fbf30 0x7fa45c10c4f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa44c004820 tx=0x7fa44c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:32:09.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:09.031+0000 7fa443fff640 1 -- 172.21.15.28:0/2202071474 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa44c032650 con 0x7fa45c0fbf30 2024-06-18T02:32:09.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:09.031+0000 7fa461e8a640 1 -- 172.21.15.28:0/2202071474 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa45c10cc30 con 0x7fa45c0fbf30 2024-06-18T02:32:09.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:09.031+0000 7fa443fff640 1 -- 172.21.15.28:0/2202071474 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa44c032c80 con 0x7fa45c0fbf30 2024-06-18T02:32:09.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:09.031+0000 7fa443fff640 1 -- 172.21.15.28:0/2202071474 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa44c03eda0 con 0x7fa45c0fbf30 2024-06-18T02:32:09.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:09.031+0000 7fa461e8a640 1 -- 172.21.15.28:0/2202071474 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa45c10d7c0 con 0x7fa45c0fbf30 2024-06-18T02:32:09.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:09.032+0000 7fa443fff640 1 -- 172.21.15.28:0/2202071474 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 16) v1 ==== 41867+0+0 (secure 0 0 0) 0x7fa44c030020 con 0x7fa45c0fbf30 2024-06-18T02:32:09.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:09.032+0000 7fa443fff640 1 --2- 172.21.15.28:0/2202071474 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fa444034790 0x7fa444036c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:32:09.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:09.033+0000 7fa461e8a640 1 -- 172.21.15.28:0/2202071474 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa45c1024a0 con 0x7fa45c0fbf30 2024-06-18T02:32:09.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:09.033+0000 7fa443fff640 1 -- 172.21.15.28:0/2202071474 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fa44c066a80 con 0x7fa45c0fbf30 2024-06-18T02:32:09.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:09.033+0000 7fa45affd640 1 --2- 172.21.15.28:0/2202071474 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fa444034790 0x7fa444036c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:32:09.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:09.033+0000 7fa45affd640 1 --2- 172.21.15.28:0/2202071474 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fa444034790 0x7fa444036c40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa450006fd0 tx=0x7fa450006db0 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:32:09.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:09.038+0000 7fa443fff640 1 -- 172.21.15.28:0/2202071474 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fa44c033420 con 0x7fa45c0fbf30 2024-06-18T02:32:09.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:09.289+0000 7fa461e8a640 1 -- 172.21.15.28:0/2202071474 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa45c10e1c0 con 0x7fa45c0fbf30 2024-06-18T02:32:09.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:09.289+0000 7fa443fff640 1 -- 172.21.15.28:0/2202071474 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+735 (secure 0 0 0) 0x7fa44c03c330 con 0x7fa45c0fbf30 2024-06-18T02:32:09.289 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:32:09.289 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":1,"fsid":"7eefb63e-2d1a-11ef-bc9d-c7b262605968","modified":"2024-06-18T02:29:55.463546Z","created":"2024-06-18T02:29:55.463546Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi001","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-18T02:32:09.290 INFO:teuthology.orchestra.run.smithi028.stderr:dumped monmap epoch 1 2024-06-18T02:32:09.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:09.291+0000 7fa461e8a640 1 -- 172.21.15.28:0/2202071474 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fa444034790 msgr2=0x7fa444036c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:32:09.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:09.291+0000 7fa461e8a640 1 --2- 172.21.15.28:0/2202071474 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fa444034790 0x7fa444036c40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa450006fd0 tx=0x7fa450006db0 comp rx=0 tx=0).stop 2024-06-18T02:32:09.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:09.291+0000 7fa461e8a640 1 -- 172.21.15.28:0/2202071474 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa45c0fbf30 msgr2=0x7fa45c10c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:32:09.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:09.291+0000 7fa461e8a640 1 --2- 172.21.15.28:0/2202071474 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa45c0fbf30 0x7fa45c10c4f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa44c004820 tx=0x7fa44c009da0 comp rx=0 tx=0).stop 2024-06-18T02:32:09.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:09.291+0000 7fa461e8a640 1 -- 172.21.15.28:0/2202071474 shutdown_connections 2024-06-18T02:32:09.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:09.291+0000 7fa461e8a640 1 --2- 172.21.15.28:0/2202071474 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fa444034790 0x7fa444036c40 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:32:09.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:09.291+0000 7fa461e8a640 1 --2- 172.21.15.28:0/2202071474 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa45c0fbf30 0x7fa45c10c4f0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:32:09.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:09.291+0000 7fa461e8a640 1 -- 172.21.15.28:0/2202071474 >> 172.21.15.28:0/2202071474 conn(0x7fa45c0f7710 msgr2=0x7fa45c1030d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:32:09.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:09.291+0000 7fa461e8a640 1 -- 172.21.15.28:0/2202071474 shutdown_connections 2024-06-18T02:32:09.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:09.291+0000 7fa461e8a640 1 -- 172.21.15.28:0/2202071474 wait complete. 2024-06-18T02:32:09.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:09 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:09.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:09 smithi001 ceph-mon[29099]: [18/Jun/2024:02:32:08] ENGINE Bus STARTING 2024-06-18T02:32:09.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:09 smithi001 ceph-mon[29099]: [18/Jun/2024:02:32:08] ENGINE Serving on https://172.21.15.1:7150 2024-06-18T02:32:09.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:09 smithi001 ceph-mon[29099]: [18/Jun/2024:02:32:08] ENGINE Bus STARTED 2024-06-18T02:32:09.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:09 smithi001 ceph-mon[29099]: [18/Jun/2024:02:32:08] ENGINE Error in HTTPServer.serve 2024-06-18T02:32:09.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: Traceback (most recent call last): 2024-06-18T02:32:09.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-06-18T02:32:09.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: self._connections.run(self.expiration_interval) 2024-06-18T02:32:09.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-06-18T02:32:09.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: self._run(expiration_interval) 2024-06-18T02:32:09.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-06-18T02:32:09.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-06-18T02:32:09.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-06-18T02:32:09.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-06-18T02:32:09.879 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-06-18T02:32:09.879 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: s = self.context.wrap_socket( 2024-06-18T02:32:09.879 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-06-18T02:32:09.879 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: return self.sslsocket_class._create( 2024-06-18T02:32:09.879 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-06-18T02:32:09.879 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: self.do_handshake() 2024-06-18T02:32:09.879 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-06-18T02:32:09.879 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: self._sslobj.do_handshake() 2024-06-18T02:32:09.879 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1129) 2024-06-18T02:32:09.879 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:09 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:09.879 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:09 smithi001 ceph-mon[29099]: from='client.? 172.21.15.28:0/2202071474' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-18T02:32:10.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:10 smithi001 ceph-mon[29099]: mgrmap e17: smithi001.teoawz(active, since 2s) 2024-06-18T02:32:11.519 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-18T02:32:11.519 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph mon dump -f json 2024-06-18T02:32:11.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:11 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:11.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:11 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:11.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:11 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:11.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:11 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:11.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:11 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config rm", "who": "osd/host:smithi001", "name": "osd_memory_target"}]: dispatch 2024-06-18T02:32:12.138 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-18T02:32:12.190 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-18T02:32:13.628 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:13 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:13.628 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:13 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:13.628 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:13 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:13.628 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:13 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:13.628 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:13 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config rm", "who": "osd/host:smithi028", "name": "osd_memory_target"}]: dispatch 2024-06-18T02:32:13.628 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:13 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:32:13.628 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:13 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T02:32:13.628 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:13 smithi001 ceph-mon[29099]: Updating smithi001:/etc/ceph/ceph.conf 2024-06-18T02:32:13.628 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:13 smithi001 ceph-mon[29099]: Updating smithi028:/etc/ceph/ceph.conf 2024-06-18T02:32:13.628 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:13 smithi001 ceph-mon[29099]: Updating smithi028:/var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-18T02:32:13.629 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:13 smithi001 ceph-mon[29099]: Updating smithi001:/var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-18T02:32:13.629 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:13 smithi001 ceph-mon[29099]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-06-18T02:32:13.629 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:13 smithi001 ceph-mon[29099]: Updating smithi001:/etc/ceph/ceph.client.admin.keyring 2024-06-18T02:32:13.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:13.738+0000 7f66b2f2d640 1 -- 172.21.15.28:0/3281352702 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66ac100e40 msgr2=0x7f66ac101210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:32:13.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:13.738+0000 7f66b2f2d640 1 --2- 172.21.15.28:0/3281352702 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66ac100e40 0x7f66ac101210 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f66a0004850 tx=0x7f66a0009da0 comp rx=0 tx=0).stop 2024-06-18T02:32:13.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:13.738+0000 7f66b2f2d640 1 -- 172.21.15.28:0/3281352702 shutdown_connections 2024-06-18T02:32:13.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:13.738+0000 7f66b2f2d640 1 --2- 172.21.15.28:0/3281352702 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66ac100e40 0x7f66ac101210 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:32:13.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:13.738+0000 7f66b2f2d640 1 -- 172.21.15.28:0/3281352702 >> 172.21.15.28:0/3281352702 conn(0x7f66ac070e10 msgr2=0x7f66ac071210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:32:13.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:13.738+0000 7f66b2f2d640 1 -- 172.21.15.28:0/3281352702 shutdown_connections 2024-06-18T02:32:13.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:13.738+0000 7f66b2f2d640 1 -- 172.21.15.28:0/3281352702 wait complete. 2024-06-18T02:32:13.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:13.739+0000 7f66b2f2d640 1 Processor -- start 2024-06-18T02:32:13.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:13.739+0000 7f66b2f2d640 1 -- start start 2024-06-18T02:32:13.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:13.739+0000 7f66b2f2d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66ac100e40 0x7f66ac0fcae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:32:13.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:13.739+0000 7f66b2f2d640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66ac0fe780 con 0x7f66ac100e40 2024-06-18T02:32:13.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:13.739+0000 7f66b0cc9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66ac100e40 0x7f66ac0fcae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:32:13.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:13.740+0000 7f66b0cc9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66ac100e40 0x7f66ac0fcae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.28:35570/0 (socket says 172.21.15.28:35570) 2024-06-18T02:32:13.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:13.740+0000 7f66b0cc9640 1 -- 172.21.15.28:0/3452791989 learned_addr learned my addr 172.21.15.28:0/3452791989 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-06-18T02:32:13.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:13.740+0000 7f66b0cc9640 1 -- 172.21.15.28:0/3452791989 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66a0005040 con 0x7f66ac100e40 2024-06-18T02:32:13.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:13.740+0000 7f66b0cc9640 1 --2- 172.21.15.28:0/3452791989 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66ac100e40 0x7f66ac0fcae0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f66a0004820 tx=0x7f66a0009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:32:13.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:13.740+0000 7f66a97fa640 1 -- 172.21.15.28:0/3452791989 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f66a002cf10 con 0x7f66ac100e40 2024-06-18T02:32:13.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:13.740+0000 7f66a97fa640 1 -- 172.21.15.28:0/3452791989 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f66a0033b00 con 0x7f66ac100e40 2024-06-18T02:32:13.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:13.740+0000 7f66b2f2d640 1 -- 172.21.15.28:0/3452791989 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66ac0fe980 con 0x7f66ac100e40 2024-06-18T02:32:13.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:13.740+0000 7f66a97fa640 1 -- 172.21.15.28:0/3452791989 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f66a003db70 con 0x7f66ac100e40 2024-06-18T02:32:13.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:13.740+0000 7f66b2f2d640 1 -- 172.21.15.28:0/3452791989 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66ac0fd1c0 con 0x7f66ac100e40 2024-06-18T02:32:13.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:13.741+0000 7f66a97fa640 1 -- 172.21.15.28:0/3452791989 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) v1 ==== 42057+0+0 (secure 0 0 0) 0x7f66a0030020 con 0x7f66ac100e40 2024-06-18T02:32:13.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:13.741+0000 7f66a97fa640 1 --2- 172.21.15.28:0/3452791989 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f669c0349f0 0x7f669c036ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:32:13.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:13.741+0000 7f66a97fa640 1 -- 172.21.15.28:0/3452791989 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f66a0066eb0 con 0x7f66ac100e40 2024-06-18T02:32:13.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:13.741+0000 7f66b2f2d640 1 -- 172.21.15.28:0/3452791989 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66ac0fc5b0 con 0x7f66ac100e40 2024-06-18T02:32:13.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:13.741+0000 7f66abfff640 1 --2- 172.21.15.28:0/3452791989 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f669c0349f0 0x7f669c036ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:32:13.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:13.742+0000 7f66abfff640 1 --2- 172.21.15.28:0/3452791989 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f669c0349f0 0x7f669c036ea0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6698006fd0 tx=0x7f6698006db0 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:32:13.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:13.746+0000 7f66a97fa640 1 -- 172.21.15.28:0/3452791989 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f66a003b780 con 0x7f66ac100e40 2024-06-18T02:32:14.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:14.002+0000 7f66b2f2d640 1 -- 172.21.15.28:0/3452791989 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f66ac05ead0 con 0x7f66ac100e40 2024-06-18T02:32:14.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:14.003+0000 7f66a97fa640 1 -- 172.21.15.28:0/3452791989 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+735 (secure 0 0 0) 0x7f66a003c3c0 con 0x7f66ac100e40 2024-06-18T02:32:14.003 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:32:14.003 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":1,"fsid":"7eefb63e-2d1a-11ef-bc9d-c7b262605968","modified":"2024-06-18T02:29:55.463546Z","created":"2024-06-18T02:29:55.463546Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi001","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-18T02:32:14.003 INFO:teuthology.orchestra.run.smithi028.stderr:dumped monmap epoch 1 2024-06-18T02:32:14.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:14.005+0000 7f66b2f2d640 1 -- 172.21.15.28:0/3452791989 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f669c0349f0 msgr2=0x7f669c036ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:32:14.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:14.005+0000 7f66b2f2d640 1 --2- 172.21.15.28:0/3452791989 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f669c0349f0 0x7f669c036ea0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6698006fd0 tx=0x7f6698006db0 comp rx=0 tx=0).stop 2024-06-18T02:32:14.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:14.005+0000 7f66b2f2d640 1 -- 172.21.15.28:0/3452791989 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66ac100e40 msgr2=0x7f66ac0fcae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:32:14.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:14.005+0000 7f66b2f2d640 1 --2- 172.21.15.28:0/3452791989 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66ac100e40 0x7f66ac0fcae0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f66a0004820 tx=0x7f66a0009da0 comp rx=0 tx=0).stop 2024-06-18T02:32:14.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:14.005+0000 7f66b2f2d640 1 -- 172.21.15.28:0/3452791989 shutdown_connections 2024-06-18T02:32:14.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:14.005+0000 7f66b2f2d640 1 --2- 172.21.15.28:0/3452791989 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f669c0349f0 0x7f669c036ea0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:32:14.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:14.005+0000 7f66b2f2d640 1 --2- 172.21.15.28:0/3452791989 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66ac100e40 0x7f66ac0fcae0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:32:14.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:14.005+0000 7f66b2f2d640 1 -- 172.21.15.28:0/3452791989 >> 172.21.15.28:0/3452791989 conn(0x7f66ac070e10 msgr2=0x7f66ac0646f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:32:14.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:14.005+0000 7f66b2f2d640 1 -- 172.21.15.28:0/3452791989 shutdown_connections 2024-06-18T02:32:14.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:14.005+0000 7f66b2f2d640 1 -- 172.21.15.28:0/3452791989 wait complete. 2024-06-18T02:32:14.627 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:14 smithi001 ceph-mon[29099]: Updating smithi028:/var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/config/ceph.client.admin.keyring 2024-06-18T02:32:14.628 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:14 smithi001 ceph-mon[29099]: Updating smithi001:/var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/config/ceph.client.admin.keyring 2024-06-18T02:32:14.628 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:14 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:14.628 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:14 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:14.628 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:14 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:14.628 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:14 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:14.628 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:14 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:14.628 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:14 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi028", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-06-18T02:32:14.628 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:14 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi028", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-06-18T02:32:14.628 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:14 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:32:14.628 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:14 smithi001 ceph-mon[29099]: Deploying daemon crash.smithi028 on smithi028 2024-06-18T02:32:14.628 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:14 smithi001 ceph-mon[29099]: from='client.? 172.21.15.28:0/3452791989' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-18T02:32:16.386 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-18T02:32:16.386 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph mon dump -f json 2024-06-18T02:32:16.650 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-18T02:32:17.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:17 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:17.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:17 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:17.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:17 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:17.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:17 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:17.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:17 smithi001 ceph-mon[29099]: Deploying daemon node-exporter.smithi028 on smithi028 2024-06-18T02:32:17.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:17 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:18.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:18.212+0000 7fa12e7ee640 1 -- 172.21.15.28:0/928765794 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa128101f40 msgr2=0x7fa128102310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:32:18.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:18.212+0000 7fa12e7ee640 1 --2- 172.21.15.28:0/928765794 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa128101f40 0x7fa128102310 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa118004850 tx=0x7fa118009da0 comp rx=0 tx=0).stop 2024-06-18T02:32:18.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:18.212+0000 7fa12e7ee640 1 -- 172.21.15.28:0/928765794 shutdown_connections 2024-06-18T02:32:18.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:18.212+0000 7fa12e7ee640 1 --2- 172.21.15.28:0/928765794 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa128101f40 0x7fa128102310 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:32:18.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:18.212+0000 7fa12e7ee640 1 -- 172.21.15.28:0/928765794 >> 172.21.15.28:0/928765794 conn(0x7fa1280f7af0 msgr2=0x7fa1280f9f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:32:18.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:18.212+0000 7fa12e7ee640 1 -- 172.21.15.28:0/928765794 shutdown_connections 2024-06-18T02:32:18.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:18.212+0000 7fa12e7ee640 1 -- 172.21.15.28:0/928765794 wait complete. 2024-06-18T02:32:18.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:18.213+0000 7fa12e7ee640 1 Processor -- start 2024-06-18T02:32:18.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:18.213+0000 7fa12e7ee640 1 -- start start 2024-06-18T02:32:18.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:18.213+0000 7fa12e7ee640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa128101f40 0x7fa1281125e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:32:18.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:18.213+0000 7fa12e7ee640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa128112b20 con 0x7fa128101f40 2024-06-18T02:32:18.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:18.213+0000 7fa127fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa128101f40 0x7fa1281125e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:32:18.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:18.213+0000 7fa127fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa128101f40 0x7fa1281125e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.28:43712/0 (socket says 172.21.15.28:43712) 2024-06-18T02:32:18.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:18.213+0000 7fa127fff640 1 -- 172.21.15.28:0/2713325274 learned_addr learned my addr 172.21.15.28:0/2713325274 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-06-18T02:32:18.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:18.214+0000 7fa127fff640 1 -- 172.21.15.28:0/2713325274 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa118005040 con 0x7fa128101f40 2024-06-18T02:32:18.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:18.214+0000 7fa127fff640 1 --2- 172.21.15.28:0/2713325274 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa128101f40 0x7fa1281125e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa118004820 tx=0x7fa118009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:32:18.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:18.214+0000 7fa124ff9640 1 -- 172.21.15.28:0/2713325274 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa11802cf10 con 0x7fa128101f40 2024-06-18T02:32:18.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:18.214+0000 7fa12e7ee640 1 -- 172.21.15.28:0/2713325274 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa128112d20 con 0x7fa128101f40 2024-06-18T02:32:18.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:18.214+0000 7fa124ff9640 1 -- 172.21.15.28:0/2713325274 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa118033b00 con 0x7fa128101f40 2024-06-18T02:32:18.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:18.214+0000 7fa124ff9640 1 -- 172.21.15.28:0/2713325274 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa11803db70 con 0x7fa128101f40 2024-06-18T02:32:18.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:18.214+0000 7fa12e7ee640 1 -- 172.21.15.28:0/2713325274 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa128113110 con 0x7fa128101f40 2024-06-18T02:32:18.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:18.215+0000 7fa124ff9640 1 -- 172.21.15.28:0/2713325274 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) v1 ==== 42057+0+0 (secure 0 0 0) 0x7fa118030020 con 0x7fa128101f40 2024-06-18T02:32:18.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:18.215+0000 7fa12e7ee640 1 -- 172.21.15.28:0/2713325274 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1280fd6b0 con 0x7fa128101f40 2024-06-18T02:32:18.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:18.215+0000 7fa124ff9640 1 --2- 172.21.15.28:0/2713325274 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fa1100349f0 0x7fa110036ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:32:18.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:18.216+0000 7fa124ff9640 1 -- 172.21.15.28:0/2713325274 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fa118066fc0 con 0x7fa128101f40 2024-06-18T02:32:18.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:18.216+0000 7fa1277fe640 1 --2- 172.21.15.28:0/2713325274 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fa1100349f0 0x7fa110036ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:32:18.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:18.216+0000 7fa1277fe640 1 --2- 172.21.15.28:0/2713325274 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fa1100349f0 0x7fa110036ea0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa11c006fd0 tx=0x7fa11c006db0 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:32:18.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:18.220+0000 7fa124ff9640 1 -- 172.21.15.28:0/2713325274 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fa11803b780 con 0x7fa128101f40 2024-06-18T02:32:18.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:18.462+0000 7fa12e7ee640 1 -- 172.21.15.28:0/2713325274 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa12804b480 con 0x7fa128101f40 2024-06-18T02:32:18.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:18.463+0000 7fa124ff9640 1 -- 172.21.15.28:0/2713325274 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+735 (secure 0 0 0) 0x7fa118044300 con 0x7fa128101f40 2024-06-18T02:32:18.463 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:32:18.463 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":1,"fsid":"7eefb63e-2d1a-11ef-bc9d-c7b262605968","modified":"2024-06-18T02:29:55.463546Z","created":"2024-06-18T02:29:55.463546Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi001","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-18T02:32:18.463 INFO:teuthology.orchestra.run.smithi028.stderr:dumped monmap epoch 1 2024-06-18T02:32:18.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:18.465+0000 7fa12e7ee640 1 -- 172.21.15.28:0/2713325274 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fa1100349f0 msgr2=0x7fa110036ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:32:18.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:18.465+0000 7fa12e7ee640 1 --2- 172.21.15.28:0/2713325274 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fa1100349f0 0x7fa110036ea0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa11c006fd0 tx=0x7fa11c006db0 comp rx=0 tx=0).stop 2024-06-18T02:32:18.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:18.465+0000 7fa12e7ee640 1 -- 172.21.15.28:0/2713325274 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa128101f40 msgr2=0x7fa1281125e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:32:18.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:18.465+0000 7fa12e7ee640 1 --2- 172.21.15.28:0/2713325274 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa128101f40 0x7fa1281125e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa118004820 tx=0x7fa118009da0 comp rx=0 tx=0).stop 2024-06-18T02:32:18.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:18.465+0000 7fa12e7ee640 1 -- 172.21.15.28:0/2713325274 shutdown_connections 2024-06-18T02:32:18.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:18.465+0000 7fa12e7ee640 1 --2- 172.21.15.28:0/2713325274 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fa1100349f0 0x7fa110036ea0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:32:18.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:18.465+0000 7fa12e7ee640 1 --2- 172.21.15.28:0/2713325274 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa128101f40 0x7fa1281125e0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:32:18.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:18.465+0000 7fa12e7ee640 1 -- 172.21.15.28:0/2713325274 >> 172.21.15.28:0/2713325274 conn(0x7fa1280f7af0 msgr2=0x7fa1280f91d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:32:18.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:18.465+0000 7fa12e7ee640 1 -- 172.21.15.28:0/2713325274 shutdown_connections 2024-06-18T02:32:18.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:18.465+0000 7fa12e7ee640 1 -- 172.21.15.28:0/2713325274 wait complete. 2024-06-18T02:32:18.779 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:18 smithi001 ceph-mon[29099]: from='client.? 172.21.15.28:0/2713325274' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-18T02:32:20.806 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-18T02:32:20.806 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph mon dump -f json 2024-06-18T02:32:21.683 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-18T02:32:22.127 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:21 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:22.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:21 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:22.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:21 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:22.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:21 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:22.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:21 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi028.xnuopx", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-18T02:32:22.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:21 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi028.xnuopx", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-06-18T02:32:22.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:21 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-18T02:32:22.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:21 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:32:22.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:21 smithi001 ceph-mon[29099]: Deploying daemon mgr.smithi028.xnuopx on smithi028 2024-06-18T02:32:22.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:21 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:32:22.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:21 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:23.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:23.665+0000 7ff459788640 1 -- 172.21.15.28:0/2781250081 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4540ff500 msgr2=0x7ff4540ff8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:32:23.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:23.665+0000 7ff459788640 1 --2- 172.21.15.28:0/2781250081 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4540ff500 0x7ff4540ff8d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ff444004850 tx=0x7ff444009da0 comp rx=0 tx=0).stop 2024-06-18T02:32:23.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:23.665+0000 7ff459788640 1 -- 172.21.15.28:0/2781250081 shutdown_connections 2024-06-18T02:32:23.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:23.665+0000 7ff459788640 1 --2- 172.21.15.28:0/2781250081 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4540ff500 0x7ff4540ff8d0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:32:23.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:23.665+0000 7ff459788640 1 -- 172.21.15.28:0/2781250081 >> 172.21.15.28:0/2781250081 conn(0x7ff454071520 msgr2=0x7ff454071920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:32:23.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:23.665+0000 7ff459788640 1 -- 172.21.15.28:0/2781250081 shutdown_connections 2024-06-18T02:32:23.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:23.665+0000 7ff459788640 1 -- 172.21.15.28:0/2781250081 wait complete. 2024-06-18T02:32:23.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:23.665+0000 7ff459788640 1 Processor -- start 2024-06-18T02:32:23.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:23.665+0000 7ff459788640 1 -- start start 2024-06-18T02:32:23.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:23.666+0000 7ff459788640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4540ff500 0x7ff45410d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:32:23.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:23.666+0000 7ff459788640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff45410db30 con 0x7ff4540ff500 2024-06-18T02:32:23.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:23.666+0000 7ff452ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4540ff500 0x7ff45410d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:32:23.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:23.666+0000 7ff452ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4540ff500 0x7ff45410d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.28:43722/0 (socket says 172.21.15.28:43722) 2024-06-18T02:32:23.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:23.666+0000 7ff452ffd640 1 -- 172.21.15.28:0/627390137 learned_addr learned my addr 172.21.15.28:0/627390137 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-06-18T02:32:23.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:23.666+0000 7ff452ffd640 1 -- 172.21.15.28:0/627390137 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff444005040 con 0x7ff4540ff500 2024-06-18T02:32:23.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:23.666+0000 7ff452ffd640 1 --2- 172.21.15.28:0/627390137 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4540ff500 0x7ff45410d5f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ff444004820 tx=0x7ff444009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:32:23.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:23.667+0000 7ff43b7fe640 1 -- 172.21.15.28:0/627390137 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff44402cf10 con 0x7ff4540ff500 2024-06-18T02:32:23.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:23.667+0000 7ff43b7fe640 1 -- 172.21.15.28:0/627390137 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff444033b00 con 0x7ff4540ff500 2024-06-18T02:32:23.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:23.667+0000 7ff43b7fe640 1 -- 172.21.15.28:0/627390137 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff44403db70 con 0x7ff4540ff500 2024-06-18T02:32:23.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:23.667+0000 7ff459788640 1 -- 172.21.15.28:0/627390137 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff45410dd30 con 0x7ff4540ff500 2024-06-18T02:32:23.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:23.667+0000 7ff459788640 1 -- 172.21.15.28:0/627390137 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff45410e1a0 con 0x7ff4540ff500 2024-06-18T02:32:23.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:23.667+0000 7ff43b7fe640 1 -- 172.21.15.28:0/627390137 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) v1 ==== 42057+0+0 (secure 0 0 0) 0x7ff444030020 con 0x7ff4540ff500 2024-06-18T02:32:23.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:23.668+0000 7ff459788640 1 -- 172.21.15.28:0/627390137 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4540f9720 con 0x7ff4540ff500 2024-06-18T02:32:23.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:23.668+0000 7ff43b7fe640 1 --2- 172.21.15.28:0/627390137 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7ff43c0349f0 0x7ff43c036ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:32:23.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:23.668+0000 7ff43b7fe640 1 -- 172.21.15.28:0/627390137 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7ff444067180 con 0x7ff4540ff500 2024-06-18T02:32:23.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:23.668+0000 7ff4527fc640 1 --2- 172.21.15.28:0/627390137 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7ff43c0349f0 0x7ff43c036ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:32:23.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:23.668+0000 7ff4527fc640 1 --2- 172.21.15.28:0/627390137 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7ff43c0349f0 0x7ff43c036ea0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff448006fd0 tx=0x7ff448006db0 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:32:23.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:23.673+0000 7ff43b7fe640 1 -- 172.21.15.28:0/627390137 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7ff44403d420 con 0x7ff4540ff500 2024-06-18T02:32:23.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:23.914+0000 7ff459788640 1 -- 172.21.15.28:0/627390137 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff45404b480 con 0x7ff4540ff500 2024-06-18T02:32:23.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:23.915+0000 7ff43b7fe640 1 -- 172.21.15.28:0/627390137 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+735 (secure 0 0 0) 0x7ff444045390 con 0x7ff4540ff500 2024-06-18T02:32:23.915 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:32:23.915 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":1,"fsid":"7eefb63e-2d1a-11ef-bc9d-c7b262605968","modified":"2024-06-18T02:29:55.463546Z","created":"2024-06-18T02:29:55.463546Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi001","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-18T02:32:23.915 INFO:teuthology.orchestra.run.smithi028.stderr:dumped monmap epoch 1 2024-06-18T02:32:23.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:23.917+0000 7ff459788640 1 -- 172.21.15.28:0/627390137 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7ff43c0349f0 msgr2=0x7ff43c036ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:32:23.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:23.917+0000 7ff459788640 1 --2- 172.21.15.28:0/627390137 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7ff43c0349f0 0x7ff43c036ea0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff448006fd0 tx=0x7ff448006db0 comp rx=0 tx=0).stop 2024-06-18T02:32:23.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:23.917+0000 7ff459788640 1 -- 172.21.15.28:0/627390137 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4540ff500 msgr2=0x7ff45410d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:32:23.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:23.917+0000 7ff459788640 1 --2- 172.21.15.28:0/627390137 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4540ff500 0x7ff45410d5f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ff444004820 tx=0x7ff444009da0 comp rx=0 tx=0).stop 2024-06-18T02:32:23.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:23.917+0000 7ff459788640 1 -- 172.21.15.28:0/627390137 shutdown_connections 2024-06-18T02:32:23.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:23.917+0000 7ff459788640 1 --2- 172.21.15.28:0/627390137 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7ff43c0349f0 0x7ff43c036ea0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:32:23.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:23.917+0000 7ff459788640 1 --2- 172.21.15.28:0/627390137 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4540ff500 0x7ff45410d5f0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:32:23.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:23.917+0000 7ff459788640 1 -- 172.21.15.28:0/627390137 >> 172.21.15.28:0/627390137 conn(0x7ff454071520 msgr2=0x7ff4540fc0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:32:23.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:23.917+0000 7ff459788640 1 -- 172.21.15.28:0/627390137 shutdown_connections 2024-06-18T02:32:23.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:23.917+0000 7ff459788640 1 -- 172.21.15.28:0/627390137 wait complete. 2024-06-18T02:32:24.377 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:23 smithi001 ceph-mon[29099]: from='client.? 172.21.15.28:0/627390137' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-18T02:32:25.765 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-18T02:32:25.765 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph mon dump -f json 2024-06-18T02:32:25.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:25 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:25.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:25 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:25.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:25 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:25.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:25 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:25.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:25 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-18T02:32:25.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:25 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:32:25.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:25 smithi001 ceph-mon[29099]: Deploying daemon mon.smithi028 on smithi028 2024-06-18T02:32:26.737 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-18T02:32:27.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:27 smithi001 ceph-mon[29099]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:32:27.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:27 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:29.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:29.110+0000 7fbfbe025640 1 -- 172.21.15.28:0/1832781541 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfb8101f40 msgr2=0x7fbfb8102310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:32:29.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:29.110+0000 7fbfbe025640 1 --2- 172.21.15.28:0/1832781541 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfb8101f40 0x7fbfb8102310 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fbfa8004850 tx=0x7fbfa8009da0 comp rx=0 tx=0).stop 2024-06-18T02:32:29.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:29.110+0000 7fbfbe025640 1 -- 172.21.15.28:0/1832781541 shutdown_connections 2024-06-18T02:32:29.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:29.110+0000 7fbfbe025640 1 --2- 172.21.15.28:0/1832781541 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfb8101f40 0x7fbfb8102310 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:32:29.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:29.110+0000 7fbfbe025640 1 -- 172.21.15.28:0/1832781541 >> 172.21.15.28:0/1832781541 conn(0x7fbfb8070e10 msgr2=0x7fbfb8071210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:32:29.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:29.110+0000 7fbfbe025640 1 -- 172.21.15.28:0/1832781541 shutdown_connections 2024-06-18T02:32:29.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:29.110+0000 7fbfbe025640 1 -- 172.21.15.28:0/1832781541 wait complete. 2024-06-18T02:32:29.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:29.110+0000 7fbfbe025640 1 Processor -- start 2024-06-18T02:32:29.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:29.111+0000 7fbfbe025640 1 -- start start 2024-06-18T02:32:29.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:29.111+0000 7fbfbe025640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfb8101f40 0x7fbfb80faec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:32:29.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:29.111+0000 7fbfbe025640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb80fb400 con 0x7fbfb8101f40 2024-06-18T02:32:29.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:29.111+0000 7fbfb77fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfb8101f40 0x7fbfb80faec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:32:29.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:29.111+0000 7fbfb77fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfb8101f40 0x7fbfb80faec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.28:39352/0 (socket says 172.21.15.28:39352) 2024-06-18T02:32:29.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:29.111+0000 7fbfb77fe640 1 -- 172.21.15.28:0/476612500 learned_addr learned my addr 172.21.15.28:0/476612500 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-06-18T02:32:29.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:29.112+0000 7fbfb77fe640 1 -- 172.21.15.28:0/476612500 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfa8005040 con 0x7fbfb8101f40 2024-06-18T02:32:29.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:29.112+0000 7fbfb77fe640 1 --2- 172.21.15.28:0/476612500 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfb8101f40 0x7fbfb80faec0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fbfa8004820 tx=0x7fbfa8009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:32:29.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:29.112+0000 7fbf9ffff640 1 -- 172.21.15.28:0/476612500 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbfa802cf10 con 0x7fbfb8101f40 2024-06-18T02:32:29.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:29.112+0000 7fbfbe025640 1 -- 172.21.15.28:0/476612500 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfb80fb600 con 0x7fbfb8101f40 2024-06-18T02:32:29.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:29.112+0000 7fbf9ffff640 1 -- 172.21.15.28:0/476612500 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbfa8033b00 con 0x7fbfb8101f40 2024-06-18T02:32:29.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:29.112+0000 7fbfbe025640 1 -- 172.21.15.28:0/476612500 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfb80fba70 con 0x7fbfb8101f40 2024-06-18T02:32:29.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:29.112+0000 7fbf9ffff640 1 -- 172.21.15.28:0/476612500 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbfa803db70 con 0x7fbfb8101f40 2024-06-18T02:32:29.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:29.113+0000 7fbf9ffff640 1 -- 172.21.15.28:0/476612500 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 17) v1 ==== 42057+0+0 (secure 0 0 0) 0x7fbfa8030020 con 0x7fbfb8101f40 2024-06-18T02:32:29.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:29.113+0000 7fbf9ffff640 1 --2- 172.21.15.28:0/476612500 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fbfa0034a40 0x7fbfa0036ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:32:29.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:29.113+0000 7fbfbe025640 1 -- 172.21.15.28:0/476612500 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfb80f9140 con 0x7fbfb8101f40 2024-06-18T02:32:29.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:29.113+0000 7fbf9ffff640 1 -- 172.21.15.28:0/476612500 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fbfa8066dc0 con 0x7fbfb8101f40 2024-06-18T02:32:29.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:29.114+0000 7fbfb6ffd640 1 --2- 172.21.15.28:0/476612500 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fbfa0034a40 0x7fbfa0036ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:32:29.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:29.114+0000 7fbfb6ffd640 1 --2- 172.21.15.28:0/476612500 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fbfa0034a40 0x7fbfa0036ef0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fbfac006fd0 tx=0x7fbfac006db0 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:32:29.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:29.118+0000 7fbf9ffff640 1 -- 172.21.15.28:0/476612500 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fbfa80463e0 con 0x7fbfb8101f40 2024-06-18T02:32:29.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:29.360+0000 7fbf9ffff640 1 -- 172.21.15.28:0/476612500 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbfa80463e0 con 0x7fbfb8101f40 2024-06-18T02:32:29.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:29.365+0000 7fbfbe025640 1 -- 172.21.15.28:0/476612500 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbfb80fd6b0 con 0x7fbfb8101f40 2024-06-18T02:32:29.524 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:29 smithi028 systemd[1]: Started Ceph mon.smithi028 for 7eefb63e-2d1a-11ef-bc9d-c7b262605968. 2024-06-18T02:32:34.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:34.370+0000 7fbf9ffff640 1 -- 172.21.15.28:0/476612500 <== mon.0 v2:172.21.15.1:3300/0 8 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1009 (secure 0 0 0) 0x7fbfa8045390 con 0x7fbfb8101f40 2024-06-18T02:32:34.371 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-06-18T02:32:34.371 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":2,"fsid":"7eefb63e-2d1a-11ef-bc9d-c7b262605968","modified":"2024-06-18T02:32:29.360970Z","created":"2024-06-18T02:29:55.463546Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi001","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi028","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-06-18T02:32:34.371 INFO:teuthology.orchestra.run.smithi028.stderr:dumped monmap epoch 2 2024-06-18T02:32:34.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:34.372+0000 7fbfbe025640 1 -- 172.21.15.28:0/476612500 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fbfa0034a40 msgr2=0x7fbfa0036ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:32:34.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:34.372+0000 7fbfbe025640 1 --2- 172.21.15.28:0/476612500 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fbfa0034a40 0x7fbfa0036ef0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fbfac006fd0 tx=0x7fbfac006db0 comp rx=0 tx=0).stop 2024-06-18T02:32:34.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:34.372+0000 7fbfbe025640 1 -- 172.21.15.28:0/476612500 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfb8101f40 msgr2=0x7fbfb80faec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:32:34.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:34.372+0000 7fbfbe025640 1 --2- 172.21.15.28:0/476612500 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfb8101f40 0x7fbfb80faec0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fbfa8004820 tx=0x7fbfa8009da0 comp rx=0 tx=0).stop 2024-06-18T02:32:34.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:34.372+0000 7fbfbe025640 1 -- 172.21.15.28:0/476612500 shutdown_connections 2024-06-18T02:32:34.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:34.372+0000 7fbfbe025640 1 --2- 172.21.15.28:0/476612500 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fbfa0034a40 0x7fbfa0036ef0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:32:34.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:34.372+0000 7fbfbe025640 1 --2- 172.21.15.28:0/476612500 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbfb8101f40 0x7fbfb80faec0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:32:34.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:34.372+0000 7fbfbe025640 1 -- 172.21.15.28:0/476612500 >> 172.21.15.28:0/476612500 conn(0x7fbfb8070e10 msgr2=0x7fbfb80f9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:32:34.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:34.372+0000 7fbfbe025640 1 -- 172.21.15.28:0/476612500 shutdown_connections 2024-06-18T02:32:34.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:32:34.372+0000 7fbfbe025640 1 -- 172.21.15.28:0/476612500 wait complete. 2024-06-18T02:32:34.732 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:34 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "mon metadata", "id": "smithi001"}]: dispatch 2024-06-18T02:32:34.732 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:34 smithi028 ceph-mon[35823]: mon.smithi001 calling monitor election 2024-06-18T02:32:34.732 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:34 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "mon metadata", "id": "smithi028"}]: dispatch 2024-06-18T02:32:34.732 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:34 smithi028 ceph-mon[35823]: from='client.? 172.21.15.28:0/476612500' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-18T02:32:34.732 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:34 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "mon metadata", "id": "smithi028"}]: dispatch 2024-06-18T02:32:34.732 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:34 smithi028 ceph-mon[35823]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:32:34.732 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:34 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "mon metadata", "id": "smithi028"}]: dispatch 2024-06-18T02:32:34.732 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:34 smithi028 ceph-mon[35823]: mon.smithi028 calling monitor election 2024-06-18T02:32:34.732 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:34 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "mon metadata", "id": "smithi028"}]: dispatch 2024-06-18T02:32:34.733 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:34 smithi028 ceph-mon[35823]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:32:34.733 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:34 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "mon metadata", "id": "smithi028"}]: dispatch 2024-06-18T02:32:34.733 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:34 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "mon metadata", "id": "smithi028"}]: dispatch 2024-06-18T02:32:34.733 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:34 smithi028 ceph-mon[35823]: mon.smithi001 is new leader, mons smithi001,smithi028 in quorum (ranks 0,1) 2024-06-18T02:32:34.733 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:34 smithi028 ceph-mon[35823]: monmap e2: 2 mons at {smithi001=[v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0],smithi028=[v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-06-18T02:32:34.733 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:34 smithi028 ceph-mon[35823]: fsmap 2024-06-18T02:32:34.733 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:34 smithi028 ceph-mon[35823]: osdmap e5: 0 total, 0 up, 0 in 2024-06-18T02:32:34.733 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:34 smithi028 ceph-mon[35823]: mgrmap e17: smithi001.teoawz(active, since 27s) 2024-06-18T02:32:34.733 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:34 smithi028 ceph-mon[35823]: Health detail: HEALTH_WARN 3 mgr modules have recently crashed 2024-06-18T02:32:34.733 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:34 smithi028 ceph-mon[35823]: [WRN] RECENT_MGR_MODULE_CRASH: 3 mgr modules have recently crashed 2024-06-18T02:32:34.733 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:34 smithi028 ceph-mon[35823]: mgr module k8sevents crashed in daemon mgr.smithi001.teoawz on host smithi001 at 2024-06-18T02:30:05.806014Z 2024-06-18T02:32:34.733 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:34 smithi028 ceph-mon[35823]: mgr module k8sevents crashed in daemon mgr.smithi001.teoawz on host smithi001 at 2024-06-18T02:30:25.506566Z 2024-06-18T02:32:34.734 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:34 smithi028 ceph-mon[35823]: mgr module k8sevents crashed in daemon mgr.smithi001.teoawz on host smithi001 at 2024-06-18T02:30:55.258856Z 2024-06-18T02:32:34.734 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:34 smithi028 ceph-mon[35823]: Standby manager daemon smithi028.xnuopx started 2024-06-18T02:32:34.734 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:34 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:34.734 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:34 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:34.734 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:34 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:34.734 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:34 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:34.734 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:34 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:34.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:34 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "mon metadata", "id": "smithi001"}]: dispatch 2024-06-18T02:32:34.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:34 smithi001 ceph-mon[29099]: mon.smithi001 calling monitor election 2024-06-18T02:32:34.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:34 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "mon metadata", "id": "smithi028"}]: dispatch 2024-06-18T02:32:34.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:34 smithi001 ceph-mon[29099]: from='client.? 172.21.15.28:0/476612500' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-18T02:32:34.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:34 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "mon metadata", "id": "smithi028"}]: dispatch 2024-06-18T02:32:34.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:34 smithi001 ceph-mon[29099]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:32:34.879 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:34 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "mon metadata", "id": "smithi028"}]: dispatch 2024-06-18T02:32:34.879 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:34 smithi001 ceph-mon[29099]: mon.smithi028 calling monitor election 2024-06-18T02:32:34.879 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:34 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "mon metadata", "id": "smithi028"}]: dispatch 2024-06-18T02:32:34.879 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:34 smithi001 ceph-mon[29099]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:32:34.879 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:34 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "mon metadata", "id": "smithi028"}]: dispatch 2024-06-18T02:32:34.879 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:34 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "mon metadata", "id": "smithi028"}]: dispatch 2024-06-18T02:32:34.879 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:34 smithi001 ceph-mon[29099]: mon.smithi001 is new leader, mons smithi001,smithi028 in quorum (ranks 0,1) 2024-06-18T02:32:34.879 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:34 smithi001 ceph-mon[29099]: monmap e2: 2 mons at {smithi001=[v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0],smithi028=[v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-06-18T02:32:34.879 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:34 smithi001 ceph-mon[29099]: fsmap 2024-06-18T02:32:34.879 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:34 smithi001 ceph-mon[29099]: osdmap e5: 0 total, 0 up, 0 in 2024-06-18T02:32:34.879 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:34 smithi001 ceph-mon[29099]: mgrmap e17: smithi001.teoawz(active, since 27s) 2024-06-18T02:32:34.880 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:34 smithi001 ceph-mon[29099]: Health detail: HEALTH_WARN 3 mgr modules have recently crashed 2024-06-18T02:32:34.880 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:34 smithi001 ceph-mon[29099]: [WRN] RECENT_MGR_MODULE_CRASH: 3 mgr modules have recently crashed 2024-06-18T02:32:34.880 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:34 smithi001 ceph-mon[29099]: mgr module k8sevents crashed in daemon mgr.smithi001.teoawz on host smithi001 at 2024-06-18T02:30:05.806014Z 2024-06-18T02:32:34.880 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:34 smithi001 ceph-mon[29099]: mgr module k8sevents crashed in daemon mgr.smithi001.teoawz on host smithi001 at 2024-06-18T02:30:25.506566Z 2024-06-18T02:32:34.880 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:34 smithi001 ceph-mon[29099]: mgr module k8sevents crashed in daemon mgr.smithi001.teoawz on host smithi001 at 2024-06-18T02:30:55.258856Z 2024-06-18T02:32:34.880 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:34 smithi001 ceph-mon[29099]: Standby manager daemon smithi028.xnuopx started 2024-06-18T02:32:34.880 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:34 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:34.880 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:34 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:34.880 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:34 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:34.880 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:34 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:34.880 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:34 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:34.997 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-06-18T02:32:34.997 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph config generate-minimal-conf 2024-06-18T02:32:35.274 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:32:35.718 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:35 smithi001 ceph-mon[29099]: mgrmap e18: smithi001.teoawz(active, since 27s), standbys: smithi028.xnuopx 2024-06-18T02:32:35.718 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:35 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "mgr metadata", "who": "smithi028.xnuopx", "id": "smithi028.xnuopx"}]: dispatch 2024-06-18T02:32:35.719 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:35 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:32:35.719 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:35 smithi001 ceph-mon[29099]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:32:35.719 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:35 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "mon metadata", "id": "smithi028"}]: dispatch 2024-06-18T02:32:35.737 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:35 smithi028 ceph-mon[35823]: mgrmap e18: smithi001.teoawz(active, since 27s), standbys: smithi028.xnuopx 2024-06-18T02:32:35.738 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:35 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "mgr metadata", "who": "smithi028.xnuopx", "id": "smithi028.xnuopx"}]: dispatch 2024-06-18T02:32:35.738 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:35 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:32:35.738 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:35 smithi028 ceph-mon[35823]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:32:35.738 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:35 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "mon metadata", "id": "smithi028"}]: dispatch 2024-06-18T02:32:36.824 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:36 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:32:36.824 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:36 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:36.824 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:36 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:36.824 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:36 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:36.824 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:36 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:32:36.825 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:36 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T02:32:36.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:36.824+0000 7fcaad7de640 1 -- 172.21.15.1:0/388353153 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcaa8100e40 msgr2=0x7fcaa8101210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:32:36.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:36.824+0000 7fcaad7de640 1 --2- 172.21.15.1:0/388353153 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcaa8100e40 0x7fcaa8101210 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fca98004850 tx=0x7fca98009da0 comp rx=0 tx=0).stop 2024-06-18T02:32:36.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:36.824+0000 7fcaad7de640 1 -- 172.21.15.1:0/388353153 shutdown_connections 2024-06-18T02:32:36.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:36.824+0000 7fcaad7de640 1 --2- 172.21.15.1:0/388353153 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcaa8100e40 0x7fcaa8101210 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:32:36.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:36.824+0000 7fcaad7de640 1 -- 172.21.15.1:0/388353153 >> 172.21.15.1:0/388353153 conn(0x7fcaa8070e10 msgr2=0x7fcaa8071210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:32:36.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:36.824+0000 7fcaad7de640 1 -- 172.21.15.1:0/388353153 shutdown_connections 2024-06-18T02:32:36.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:36.824+0000 7fcaad7de640 1 -- 172.21.15.1:0/388353153 wait complete. 2024-06-18T02:32:36.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:36.825+0000 7fcaad7de640 1 Processor -- start 2024-06-18T02:32:36.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:36.825+0000 7fcaad7de640 1 -- start start 2024-06-18T02:32:36.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:36.826+0000 7fcaad7de640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcaa8100e40 0x7fcaa8112670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:32:36.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:36.826+0000 7fcaad7de640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcaa8112bb0 0x7fcaa810c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:32:36.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:36.826+0000 7fcaad7de640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaa81131c0 con 0x7fcaa8112bb0 2024-06-18T02:32:36.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:36.826+0000 7fcaad7de640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaa8113300 con 0x7fcaa8100e40 2024-06-18T02:32:36.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:36.826+0000 7fca9e7fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcaa8112bb0 0x7fcaa810c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:32:36.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:36.826+0000 7fca9e7fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcaa8112bb0 0x7fcaa810c630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47836/0 (socket says 172.21.15.1:47836) 2024-06-18T02:32:36.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:36.826+0000 7fca9e7fc640 1 -- 172.21.15.1:0/3312314671 learned_addr learned my addr 172.21.15.1:0/3312314671 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:32:36.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:36.826+0000 7fcaa6ffd640 1 --2- 172.21.15.1:0/3312314671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcaa8100e40 0x7fcaa8112670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:32:36.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:36.826+0000 7fcaa6ffd640 1 -- 172.21.15.1:0/3312314671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcaa8100e40 msgr2=0x7fcaa8112670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-06-18T02:32:36.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:36.826+0000 7fcaa6ffd640 1 -- 172.21.15.1:0/3312314671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcaa8100e40 msgr2=0x7fcaa8112670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-06-18T02:32:36.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:36.826+0000 7fcaa6ffd640 1 --2- 172.21.15.1:0/3312314671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcaa8100e40 0x7fcaa8112670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-06-18T02:32:36.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:36.826+0000 7fcaa6ffd640 1 --2- 172.21.15.1:0/3312314671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcaa8100e40 0x7fcaa8112670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-18T02:32:36.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:36.826+0000 7fca9e7fc640 1 -- 172.21.15.1:0/3312314671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcaa8100e40 msgr2=0x7fcaa8112670 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-18T02:32:36.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:36.826+0000 7fca9e7fc640 1 --2- 172.21.15.1:0/3312314671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcaa8100e40 0x7fcaa8112670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:32:36.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:36.826+0000 7fca9e7fc640 1 -- 172.21.15.1:0/3312314671 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca98005040 con 0x7fcaa8112bb0 2024-06-18T02:32:36.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:36.826+0000 7fca9e7fc640 1 --2- 172.21.15.1:0/3312314671 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcaa8112bb0 0x7fcaa810c630 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fca9000b790 tx=0x7fca900091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:32:36.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:36.826+0000 7fca9ffff640 1 -- 172.21.15.1:0/3312314671 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fca9000c910 con 0x7fcaa8112bb0 2024-06-18T02:32:36.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:36.826+0000 7fcaad7de640 1 -- 172.21.15.1:0/3312314671 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcaa810cc30 con 0x7fcaa8112bb0 2024-06-18T02:32:36.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:36.826+0000 7fca9ffff640 1 -- 172.21.15.1:0/3312314671 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fca9000be60 con 0x7fcaa8112bb0 2024-06-18T02:32:36.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:36.826+0000 7fca9ffff640 1 -- 172.21.15.1:0/3312314671 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fca90014ea0 con 0x7fcaa8112bb0 2024-06-18T02:32:36.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:36.827+0000 7fcaad7de640 1 -- 172.21.15.1:0/3312314671 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcaa810d150 con 0x7fcaa8112bb0 2024-06-18T02:32:36.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:36.828+0000 7fcaad7de640 1 -- 172.21.15.1:0/3312314671 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcaa810d480 con 0x7fcaa8112bb0 2024-06-18T02:32:36.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:36.828+0000 7fca9ffff640 1 -- 172.21.15.1:0/3312314671 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 18) v1 ==== 83687+0+0 (secure 0 0 0) 0x7fca90014440 con 0x7fcaa8112bb0 2024-06-18T02:32:36.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:36.829+0000 7fca9ffff640 1 --2- 172.21.15.1:0/3312314671 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fca940654b0 0x7fca94067960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:32:36.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:36.829+0000 7fca9ffff640 1 -- 172.21.15.1:0/3312314671 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fca90080e10 con 0x7fcaa8112bb0 2024-06-18T02:32:36.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:36.829+0000 7fcaa6ffd640 1 --2- 172.21.15.1:0/3312314671 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fca940654b0 0x7fca94067960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:32:36.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:36.830+0000 7fcaa6ffd640 1 --2- 172.21.15.1:0/3312314671 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fca940654b0 0x7fca94067960 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fca98009f80 tx=0x7fca98009da0 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:32:36.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:36.833+0000 7fca9ffff640 1 -- 172.21.15.1:0/3312314671 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fca90050110 con 0x7fcaa8112bb0 2024-06-18T02:32:36.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:36 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:32:36.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:36 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:36.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:36 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:36.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:36 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:36.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:36 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:32:36.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:36 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T02:32:37.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:37.042+0000 7fcaad7de640 1 -- 172.21.15.1:0/3312314671 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fcaa810dd20 con 0x7fcaa8112bb0 2024-06-18T02:32:37.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:37.043+0000 7fca9ffff640 1 -- 172.21.15.1:0/3312314671 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v10) v1 ==== 76+0+221 (secure 0 0 0) 0x7fca90019090 con 0x7fcaa8112bb0 2024-06-18T02:32:37.043 INFO:teuthology.orchestra.run.smithi001.stdout:# minimal ceph.conf for 7eefb63e-2d1a-11ef-bc9d-c7b262605968 2024-06-18T02:32:37.043 INFO:teuthology.orchestra.run.smithi001.stdout:[global] 2024-06-18T02:32:37.043 INFO:teuthology.orchestra.run.smithi001.stdout: fsid = 7eefb63e-2d1a-11ef-bc9d-c7b262605968 2024-06-18T02:32:37.043 INFO:teuthology.orchestra.run.smithi001.stdout: mon_host = [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] 2024-06-18T02:32:37.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:37.045+0000 7fcaad7de640 1 -- 172.21.15.1:0/3312314671 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fca940654b0 msgr2=0x7fca94067960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:32:37.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:37.045+0000 7fcaad7de640 1 --2- 172.21.15.1:0/3312314671 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fca940654b0 0x7fca94067960 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fca98009f80 tx=0x7fca98009da0 comp rx=0 tx=0).stop 2024-06-18T02:32:37.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:37.045+0000 7fcaad7de640 1 -- 172.21.15.1:0/3312314671 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcaa8112bb0 msgr2=0x7fcaa810c630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:32:37.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:37.045+0000 7fcaad7de640 1 --2- 172.21.15.1:0/3312314671 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcaa8112bb0 0x7fcaa810c630 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fca9000b790 tx=0x7fca900091d0 comp rx=0 tx=0).stop 2024-06-18T02:32:37.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:37.045+0000 7fcaad7de640 1 -- 172.21.15.1:0/3312314671 shutdown_connections 2024-06-18T02:32:37.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:37.045+0000 7fcaad7de640 1 --2- 172.21.15.1:0/3312314671 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fca940654b0 0x7fca94067960 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:32:37.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:37.045+0000 7fcaad7de640 1 --2- 172.21.15.1:0/3312314671 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcaa8112bb0 0x7fcaa810c630 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:32:37.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:37.045+0000 7fcaad7de640 1 --2- 172.21.15.1:0/3312314671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcaa8100e40 0x7fcaa8112670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:32:37.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:37.045+0000 7fcaad7de640 1 -- 172.21.15.1:0/3312314671 >> 172.21.15.1:0/3312314671 conn(0x7fcaa8070e10 msgr2=0x7fcaa80648b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:32:37.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:37.045+0000 7fcaad7de640 1 -- 172.21.15.1:0/3312314671 shutdown_connections 2024-06-18T02:32:37.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:37.045+0000 7fcaad7de640 1 -- 172.21.15.1:0/3312314671 wait complete. 2024-06-18T02:32:37.630 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-06-18T02:32:37.630 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-06-18T02:32:37.630 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/etc/ceph/ceph.conf 2024-06-18T02:32:37.661 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-06-18T02:32:37.661 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-06-18T02:32:37.727 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-06-18T02:32:37.727 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/ceph/ceph.conf 2024-06-18T02:32:37.759 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-06-18T02:32:37.760 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-06-18T02:32:37.828 INFO:tasks.cephadm:Deploying OSDs... 2024-06-18T02:32:37.829 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-06-18T02:32:37.829 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/scratch_devs of=/dev/stdout 2024-06-18T02:32:37.845 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-06-18T02:32:37.845 DEBUG:teuthology.orchestra.run.smithi001:> stat /dev/vg_nvme/lv_1 2024-06-18T02:32:37.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:37 smithi001 ceph-mon[29099]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:32:37.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:37 smithi001 ceph-mon[29099]: Updating smithi001:/etc/ceph/ceph.conf 2024-06-18T02:32:37.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:37 smithi001 ceph-mon[29099]: Updating smithi028:/etc/ceph/ceph.conf 2024-06-18T02:32:37.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:37 smithi001 ceph-mon[29099]: Updating smithi028:/var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-18T02:32:37.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:37 smithi001 ceph-mon[29099]: Updating smithi001:/var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-18T02:32:37.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:37 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/3312314671' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:32:37.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:37 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:37.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:37 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:37.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:37 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:37.879 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:37 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:37.879 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:37 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:37.879 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:37 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-18T02:32:37.879 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:37 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-18T02:32:37.879 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:37 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:32:37.895 INFO:teuthology.orchestra.run.smithi001.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-06-18T02:32:37.895 INFO:teuthology.orchestra.run.smithi001.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-18T02:32:37.895 INFO:teuthology.orchestra.run.smithi001.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-06-18T02:32:37.896 INFO:teuthology.orchestra.run.smithi001.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-18T02:32:37.896 INFO:teuthology.orchestra.run.smithi001.stdout:Context: system_u:object_r:device_t:s0 2024-06-18T02:32:37.896 INFO:teuthology.orchestra.run.smithi001.stdout:Access: 2024-06-18 02:32:00.214475687 +0000 2024-06-18T02:32:37.896 INFO:teuthology.orchestra.run.smithi001.stdout:Modify: 2024-06-18 02:25:37.454212515 +0000 2024-06-18T02:32:37.896 INFO:teuthology.orchestra.run.smithi001.stdout:Change: 2024-06-18 02:25:37.454212515 +0000 2024-06-18T02:32:37.896 INFO:teuthology.orchestra.run.smithi001.stdout: Birth: 2024-06-18 02:25:37.454212515 +0000 2024-06-18T02:32:37.896 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-06-18T02:32:37.960 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records in 2024-06-18T02:32:37.960 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records out 2024-06-18T02:32:37.960 INFO:teuthology.orchestra.run.smithi001.stderr:512 bytes copied, 9.2535e-05 s, 5.5 MB/s 2024-06-18T02:32:37.962 DEBUG:teuthology.orchestra.run.smithi001:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-06-18T02:32:37.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:37 smithi028 ceph-mon[35823]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:32:37.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:37 smithi028 ceph-mon[35823]: Updating smithi001:/etc/ceph/ceph.conf 2024-06-18T02:32:37.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:37 smithi028 ceph-mon[35823]: Updating smithi028:/etc/ceph/ceph.conf 2024-06-18T02:32:37.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:37 smithi028 ceph-mon[35823]: Updating smithi028:/var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-18T02:32:37.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:37 smithi028 ceph-mon[35823]: Updating smithi001:/var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-18T02:32:37.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:37 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/3312314671' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:32:37.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:37 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:37.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:37 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:37.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:37 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:37.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:37 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:37.989 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:37 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:37.989 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:37 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-18T02:32:37.989 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:37 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-18T02:32:37.989 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:37 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:32:38.019 DEBUG:teuthology.orchestra.run.smithi001:> stat /dev/vg_nvme/lv_2 2024-06-18T02:32:38.077 INFO:teuthology.orchestra.run.smithi001.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-06-18T02:32:38.077 INFO:teuthology.orchestra.run.smithi001.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-18T02:32:38.077 INFO:teuthology.orchestra.run.smithi001.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-06-18T02:32:38.077 INFO:teuthology.orchestra.run.smithi001.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-18T02:32:38.077 INFO:teuthology.orchestra.run.smithi001.stdout:Context: system_u:object_r:device_t:s0 2024-06-18T02:32:38.077 INFO:teuthology.orchestra.run.smithi001.stdout:Access: 2024-06-18 02:32:00.214475687 +0000 2024-06-18T02:32:38.077 INFO:teuthology.orchestra.run.smithi001.stdout:Modify: 2024-06-18 02:25:37.786208650 +0000 2024-06-18T02:32:38.077 INFO:teuthology.orchestra.run.smithi001.stdout:Change: 2024-06-18 02:25:37.786208650 +0000 2024-06-18T02:32:38.077 INFO:teuthology.orchestra.run.smithi001.stdout: Birth: 2024-06-18 02:25:37.786208650 +0000 2024-06-18T02:32:38.078 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-06-18T02:32:38.141 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records in 2024-06-18T02:32:38.141 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records out 2024-06-18T02:32:38.141 INFO:teuthology.orchestra.run.smithi001.stderr:512 bytes copied, 8.6769e-05 s, 5.9 MB/s 2024-06-18T02:32:38.143 DEBUG:teuthology.orchestra.run.smithi001:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-06-18T02:32:38.199 DEBUG:teuthology.orchestra.run.smithi001:> stat /dev/vg_nvme/lv_3 2024-06-18T02:32:38.256 INFO:teuthology.orchestra.run.smithi001.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-06-18T02:32:38.256 INFO:teuthology.orchestra.run.smithi001.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-18T02:32:38.256 INFO:teuthology.orchestra.run.smithi001.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-06-18T02:32:38.256 INFO:teuthology.orchestra.run.smithi001.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-18T02:32:38.256 INFO:teuthology.orchestra.run.smithi001.stdout:Context: system_u:object_r:device_t:s0 2024-06-18T02:32:38.256 INFO:teuthology.orchestra.run.smithi001.stdout:Access: 2024-06-18 02:32:00.214475687 +0000 2024-06-18T02:32:38.256 INFO:teuthology.orchestra.run.smithi001.stdout:Modify: 2024-06-18 02:25:38.120204761 +0000 2024-06-18T02:32:38.257 INFO:teuthology.orchestra.run.smithi001.stdout:Change: 2024-06-18 02:25:38.120204761 +0000 2024-06-18T02:32:38.257 INFO:teuthology.orchestra.run.smithi001.stdout: Birth: 2024-06-18 02:25:38.120204761 +0000 2024-06-18T02:32:38.257 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-06-18T02:32:38.322 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records in 2024-06-18T02:32:38.322 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records out 2024-06-18T02:32:38.322 INFO:teuthology.orchestra.run.smithi001.stderr:512 bytes copied, 9.3167e-05 s, 5.5 MB/s 2024-06-18T02:32:38.323 DEBUG:teuthology.orchestra.run.smithi001:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-06-18T02:32:38.380 DEBUG:teuthology.orchestra.run.smithi001:> stat /dev/vg_nvme/lv_4 2024-06-18T02:32:38.436 INFO:teuthology.orchestra.run.smithi001.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-06-18T02:32:38.436 INFO:teuthology.orchestra.run.smithi001.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-18T02:32:38.436 INFO:teuthology.orchestra.run.smithi001.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-06-18T02:32:38.436 INFO:teuthology.orchestra.run.smithi001.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-18T02:32:38.436 INFO:teuthology.orchestra.run.smithi001.stdout:Context: system_u:object_r:device_t:s0 2024-06-18T02:32:38.436 INFO:teuthology.orchestra.run.smithi001.stdout:Access: 2024-06-18 02:32:00.215475676 +0000 2024-06-18T02:32:38.436 INFO:teuthology.orchestra.run.smithi001.stdout:Modify: 2024-06-18 02:25:38.476200615 +0000 2024-06-18T02:32:38.436 INFO:teuthology.orchestra.run.smithi001.stdout:Change: 2024-06-18 02:25:38.476200615 +0000 2024-06-18T02:32:38.437 INFO:teuthology.orchestra.run.smithi001.stdout: Birth: 2024-06-18 02:25:38.476200615 +0000 2024-06-18T02:32:38.437 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-06-18T02:32:38.500 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records in 2024-06-18T02:32:38.500 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records out 2024-06-18T02:32:38.501 INFO:teuthology.orchestra.run.smithi001.stderr:512 bytes copied, 0.000107259 s, 4.8 MB/s 2024-06-18T02:32:38.502 DEBUG:teuthology.orchestra.run.smithi001:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-06-18T02:32:38.560 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-06-18T02:32:38.560 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/scratch_devs of=/dev/stdout 2024-06-18T02:32:38.580 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-06-18T02:32:38.580 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_1 2024-06-18T02:32:38.640 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-06-18T02:32:38.640 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-18T02:32:38.640 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-06-18T02:32:38.640 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-18T02:32:38.640 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-06-18T02:32:38.640 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-06-18 02:32:27.507112015 +0000 2024-06-18T02:32:38.640 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-06-18 02:25:44.352195036 +0000 2024-06-18T02:32:38.640 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-06-18 02:25:44.352195036 +0000 2024-06-18T02:32:38.640 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-06-18 02:25:44.352195036 +0000 2024-06-18T02:32:38.641 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-06-18T02:32:38.709 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-06-18T02:32:38.709 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-06-18T02:32:38.710 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.00013624 s, 3.8 MB/s 2024-06-18T02:32:38.711 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-06-18T02:32:38.739 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:38 smithi001 ceph-mon[29099]: Reconfiguring mon.smithi001 (unknown last config time)... 2024-06-18T02:32:38.739 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:38 smithi001 ceph-mon[29099]: Reconfiguring daemon mon.smithi001 on smithi001 2024-06-18T02:32:38.771 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_2 2024-06-18T02:32:38.830 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-06-18T02:32:38.830 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-18T02:32:38.830 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-06-18T02:32:38.830 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-18T02:32:38.831 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-06-18T02:32:38.831 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-06-18 02:32:27.507112015 +0000 2024-06-18T02:32:38.831 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-06-18 02:25:44.683185857 +0000 2024-06-18T02:32:38.831 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-06-18 02:25:44.683185857 +0000 2024-06-18T02:32:38.831 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-06-18 02:25:44.683185857 +0000 2024-06-18T02:32:38.831 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-06-18T02:32:38.896 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:38 smithi028 ceph-mon[35823]: Reconfiguring mon.smithi001 (unknown last config time)... 2024-06-18T02:32:38.897 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:38 smithi028 ceph-mon[35823]: Reconfiguring daemon mon.smithi001 on smithi001 2024-06-18T02:32:38.899 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-06-18T02:32:38.899 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-06-18T02:32:38.899 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000109329 s, 4.7 MB/s 2024-06-18T02:32:38.901 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-06-18T02:32:38.962 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_3 2024-06-18T02:32:39.021 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-06-18T02:32:39.021 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-18T02:32:39.021 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-06-18T02:32:39.021 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-18T02:32:39.021 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-06-18T02:32:39.021 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-06-18 02:32:27.508111987 +0000 2024-06-18T02:32:39.021 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-06-18 02:25:45.024176401 +0000 2024-06-18T02:32:39.021 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-06-18 02:25:45.024176401 +0000 2024-06-18T02:32:39.022 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-06-18 02:25:45.024176401 +0000 2024-06-18T02:32:39.022 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-06-18T02:32:39.090 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-06-18T02:32:39.090 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-06-18T02:32:39.091 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000125102 s, 4.1 MB/s 2024-06-18T02:32:39.092 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-06-18T02:32:39.152 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_4 2024-06-18T02:32:39.211 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-06-18T02:32:39.212 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-18T02:32:39.212 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-06-18T02:32:39.212 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-18T02:32:39.212 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-06-18T02:32:39.212 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-06-18 02:32:27.508111987 +0000 2024-06-18T02:32:39.212 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-06-18 02:25:45.362167027 +0000 2024-06-18T02:32:39.212 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-06-18 02:25:45.362167027 +0000 2024-06-18T02:32:39.212 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-06-18 02:25:45.362167027 +0000 2024-06-18T02:32:39.212 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-06-18T02:32:39.281 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-06-18T02:32:39.281 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-06-18T02:32:39.281 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000126547 s, 4.0 MB/s 2024-06-18T02:32:39.282 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-06-18T02:32:39.342 INFO:tasks.cephadm:Deploying osd.0 on smithi001 with /dev/vg_nvme/lv_4... 2024-06-18T02:32:39.342 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-06-18T02:32:39.706 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:32:40.127 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:39 smithi001 ceph-mon[29099]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:32:40.127 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:39 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:40.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:39 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:40.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:39 smithi001 ceph-mon[29099]: Reconfiguring mgr.smithi001.teoawz (unknown last config time)... 2024-06-18T02:32:40.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:39 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi001.teoawz", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-18T02:32:40.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:39 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-18T02:32:40.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:39 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:32:40.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:39 smithi001 ceph-mon[29099]: Reconfiguring daemon mgr.smithi001.teoawz on smithi001 2024-06-18T02:32:40.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:39 smithi028 ceph-mon[35823]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:32:40.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:39 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:40.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:39 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:40.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:39 smithi028 ceph-mon[35823]: Reconfiguring mgr.smithi001.teoawz (unknown last config time)... 2024-06-18T02:32:40.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:39 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi001.teoawz", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-18T02:32:40.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:39 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-18T02:32:40.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:39 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:32:40.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:39 smithi028 ceph-mon[35823]: Reconfiguring daemon mgr.smithi001.teoawz on smithi001 2024-06-18T02:32:41.096 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:41 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:41.096 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:41 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:41.097 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:41 smithi001 ceph-mon[29099]: Reconfiguring crash.smithi001 (monmap changed)... 2024-06-18T02:32:41.097 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:41 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi001", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-06-18T02:32:41.097 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:41 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:32:41.097 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:41 smithi001 ceph-mon[29099]: Reconfiguring daemon crash.smithi001 on smithi001 2024-06-18T02:32:41.097 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:41 smithi001 ceph-mon[29099]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:32:41.487 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:41 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:41.487 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:41 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:41.488 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:41 smithi028 ceph-mon[35823]: Reconfiguring crash.smithi001 (monmap changed)... 2024-06-18T02:32:41.488 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:41 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi001", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-06-18T02:32:41.488 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:41 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:32:41.488 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:41 smithi028 ceph-mon[35823]: Reconfiguring daemon crash.smithi001 on smithi001 2024-06-18T02:32:41.488 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:41 smithi028 ceph-mon[35823]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:32:42.409 INFO:teuthology.orchestra.run.smithi001.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-06-18T02:32:42.496 INFO:teuthology.orchestra.run.smithi001.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-06-18T02:32:42.506 INFO:teuthology.orchestra.run.smithi001.stderr: stderr: 10+0 records in 2024-06-18T02:32:42.506 INFO:teuthology.orchestra.run.smithi001.stderr:10+0 records out 2024-06-18T02:32:42.506 INFO:teuthology.orchestra.run.smithi001.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00841778 s, 1.2 GB/s 2024-06-18T02:32:42.507 INFO:teuthology.orchestra.run.smithi001.stderr:--> Zapping successful for: 2024-06-18T02:32:43.148 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph orch daemon add osd smithi001:vg_nvme/lv_4 2024-06-18T02:32:43.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:43 smithi001 ceph-mon[29099]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:32:43.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:43 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:43.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:43 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:43.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:43 smithi001 ceph-mon[29099]: Reconfiguring alertmanager.smithi001 (dependencies changed)... 2024-06-18T02:32:43.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:43 smithi001 ceph-mon[29099]: Reconfiguring daemon alertmanager.smithi001 on smithi001 2024-06-18T02:32:43.971 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:32:43.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:43 smithi028 ceph-mon[35823]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:32:43.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:43 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:43.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:43 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:43.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:43 smithi028 ceph-mon[35823]: Reconfiguring alertmanager.smithi001 (dependencies changed)... 2024-06-18T02:32:43.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:43 smithi028 ceph-mon[35823]: Reconfiguring daemon alertmanager.smithi001 on smithi001 2024-06-18T02:32:45.974 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:45 smithi001 ceph-mon[29099]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:32:45.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:45 smithi028 ceph-mon[35823]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:32:46.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:46.862+0000 7f306d56a640 1 -- 172.21.15.1:0/1148920079 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30680fd0b0 msgr2=0x7f30680fb4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:32:46.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:46.862+0000 7f306d56a640 1 --2- 172.21.15.1:0/1148920079 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30680fd0b0 0x7f30680fb4b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f305c0048d0 tx=0x7f305c009da0 comp rx=0 tx=0).stop 2024-06-18T02:32:46.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:46.863+0000 7f306d56a640 1 -- 172.21.15.1:0/1148920079 shutdown_connections 2024-06-18T02:32:46.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:46.863+0000 7f306d56a640 1 --2- 172.21.15.1:0/1148920079 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30680fd0b0 0x7f30680fb4b0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:32:46.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:46.863+0000 7f306d56a640 1 --2- 172.21.15.1:0/1148920079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30680fc760 0x7f30680fcb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:32:46.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:46.863+0000 7f306d56a640 1 -- 172.21.15.1:0/1148920079 >> 172.21.15.1:0/1148920079 conn(0x7f30680f60f0 msgr2=0x7f30680f8500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:32:46.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:46.863+0000 7f306d56a640 1 -- 172.21.15.1:0/1148920079 shutdown_connections 2024-06-18T02:32:46.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:46.863+0000 7f306d56a640 1 -- 172.21.15.1:0/1148920079 wait complete. 2024-06-18T02:32:46.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:46.863+0000 7f306d56a640 1 Processor -- start 2024-06-18T02:32:46.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:46.864+0000 7f306d56a640 1 -- start start 2024-06-18T02:32:46.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:46.864+0000 7f306d56a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30680fc760 0x7f306810be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:32:46.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:46.864+0000 7f306d56a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30680fd0b0 0x7f306810c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:32:46.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:46.864+0000 7f306d56a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f306810c9f0 con 0x7f30680fd0b0 2024-06-18T02:32:46.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:46.864+0000 7f306d56a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f306810d130 con 0x7f30680fc760 2024-06-18T02:32:46.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:46.864+0000 7f30667fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30680fd0b0 0x7f306810c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:32:46.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:46.864+0000 7f30667fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30680fd0b0 0x7f306810c3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47390/0 (socket says 172.21.15.1:47390) 2024-06-18T02:32:46.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:46.864+0000 7f30667fc640 1 -- 172.21.15.1:0/2533817626 learned_addr learned my addr 172.21.15.1:0/2533817626 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:32:46.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:46.864+0000 7f3066ffd640 1 --2- 172.21.15.1:0/2533817626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30680fc760 0x7f306810be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:32:46.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:46.864+0000 7f30667fc640 1 -- 172.21.15.1:0/2533817626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30680fc760 msgr2=0x7f306810be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:32:46.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:46.864+0000 7f30667fc640 1 --2- 172.21.15.1:0/2533817626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30680fc760 0x7f306810be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:32:46.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:46.864+0000 7f30667fc640 1 -- 172.21.15.1:0/2533817626 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f305c005040 con 0x7f30680fd0b0 2024-06-18T02:32:46.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:46.864+0000 7f30667fc640 1 --2- 172.21.15.1:0/2533817626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30680fd0b0 0x7f306810c3a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f305c000c00 tx=0x7f305c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:32:46.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:46.864+0000 7f304f7fe640 1 -- 172.21.15.1:0/2533817626 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f305c038070 con 0x7f30680fd0b0 2024-06-18T02:32:46.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:46.864+0000 7f306d56a640 1 -- 172.21.15.1:0/2533817626 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f306810d360 con 0x7f30680fd0b0 2024-06-18T02:32:46.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:46.864+0000 7f304f7fe640 1 -- 172.21.15.1:0/2533817626 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f305c032970 con 0x7f30680fd0b0 2024-06-18T02:32:46.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:46.864+0000 7f304f7fe640 1 -- 172.21.15.1:0/2533817626 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f305c033950 con 0x7f30680fd0b0 2024-06-18T02:32:46.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:46.865+0000 7f306d56a640 1 -- 172.21.15.1:0/2533817626 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f306810d8a0 con 0x7f30680fd0b0 2024-06-18T02:32:46.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:46.866+0000 7f306d56a640 1 -- 172.21.15.1:0/2533817626 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30680f9d70 con 0x7f30680fd0b0 2024-06-18T02:32:46.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:46.866+0000 7f304f7fe640 1 -- 172.21.15.1:0/2533817626 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 18) v1 ==== 83687+0+0 (secure 0 0 0) 0x7f305c030020 con 0x7f30680fd0b0 2024-06-18T02:32:46.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:46.866+0000 7f304f7fe640 1 --2- 172.21.15.1:0/2533817626 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f30500653a0 0x7f3050067850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:32:46.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:46.866+0000 7f304f7fe640 1 -- 172.21.15.1:0/2533817626 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f305c0a1770 con 0x7f30680fd0b0 2024-06-18T02:32:46.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:46.866+0000 7f3066ffd640 1 --2- 172.21.15.1:0/2533817626 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f30500653a0 0x7f3050067850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:32:46.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:46.867+0000 7f3066ffd640 1 --2- 172.21.15.1:0/2533817626 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f30500653a0 0x7f3050067850 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3058004550 tx=0x7f3058009040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:32:46.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:46.871+0000 7f304f7fe640 1 -- 172.21.15.1:0/2533817626 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f305c0709c0 con 0x7f30680fd0b0 2024-06-18T02:32:47.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:32:47.068+0000 7f306d56a640 1 -- 172.21.15.1:0/2533817626 --> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi001:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f306810dc70 con 0x7f30500653a0 2024-06-18T02:32:47.847 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:47 smithi001 ceph-mon[29099]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:32:47.847 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:47 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-18T02:32:47.848 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:47 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-18T02:32:47.848 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:47 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:32:47.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:47 smithi028 ceph-mon[35823]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:32:47.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:47 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-18T02:32:47.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:47 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-18T02:32:47.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:47 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:32:48.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:48 smithi028 ceph-mon[35823]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi001:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-18T02:32:49.077 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:48 smithi001 ceph-mon[29099]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi001:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-18T02:32:49.776 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:49 smithi001 ceph-mon[29099]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:32:49.776 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:49 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:49.776 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:49 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:49.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:49 smithi028 ceph-mon[35823]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:32:49.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:49 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:49.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:49 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:32:50.930 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:50 smithi001 ceph-mon[29099]: Reconfiguring grafana.smithi001 (dependencies changed)... 2024-06-18T02:32:50.930 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:50 smithi001 ceph-mon[29099]: Reconfiguring daemon grafana.smithi001 on smithi001 2024-06-18T02:32:50.930 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:50 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/3543525042' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2fc85253-858e-496d-a262-061526504dc4"}]: dispatch 2024-06-18T02:32:50.930 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:50 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/3543525042' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2fc85253-858e-496d-a262-061526504dc4"}]': finished 2024-06-18T02:32:50.930 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:50 smithi001 ceph-mon[29099]: osdmap e6: 1 total, 0 up, 1 in 2024-06-18T02:32:50.930 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:50 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-18T02:32:50.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:50 smithi028 ceph-mon[35823]: Reconfiguring grafana.smithi001 (dependencies changed)... 2024-06-18T02:32:50.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:50 smithi028 ceph-mon[35823]: Reconfiguring daemon grafana.smithi001 on smithi001 2024-06-18T02:32:50.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:50 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/3543525042' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2fc85253-858e-496d-a262-061526504dc4"}]: dispatch 2024-06-18T02:32:50.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:50 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/3543525042' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2fc85253-858e-496d-a262-061526504dc4"}]': finished 2024-06-18T02:32:50.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:50 smithi028 ceph-mon[35823]: osdmap e6: 1 total, 0 up, 1 in 2024-06-18T02:32:50.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:50 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-18T02:32:51.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:51 smithi001 ceph-mon[29099]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:32:51.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:51 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/3892124611' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-18T02:32:51.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:51 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:32:51.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:51 smithi028 ceph-mon[35823]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:32:51.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:51 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/3892124611' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-18T02:32:51.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:51 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:32:53.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:53 smithi028 ceph-mon[35823]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:32:54.106 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:53 smithi001 ceph-mon[29099]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:32:55.939 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:55 smithi001 ceph-mon[29099]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:32:55.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:55 smithi028 ceph-mon[35823]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:32:57.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:57 smithi001 ceph-mon[29099]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:32:57.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:57 smithi028 ceph-mon[35823]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:32:59.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:32:59 smithi028 ceph-mon[35823]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:33:00.001 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:32:59 smithi001 ceph-mon[29099]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:33:00.977 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:00 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:00.977 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:00 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:00.977 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:00 smithi001 ceph-mon[29099]: Reconfiguring prometheus.smithi001 (dependencies changed)... 2024-06-18T02:33:00.977 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:00 smithi001 ceph-mon[29099]: Reconfiguring daemon prometheus.smithi001 on smithi001 2024-06-18T02:33:00.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:00 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:00.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:00 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:00.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:00 smithi028 ceph-mon[35823]: Reconfiguring prometheus.smithi001 (dependencies changed)... 2024-06-18T02:33:00.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:00 smithi028 ceph-mon[35823]: Reconfiguring daemon prometheus.smithi001 on smithi001 2024-06-18T02:33:01.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:01 smithi001 ceph-mon[29099]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:33:01.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:01 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-06-18T02:33:01.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:01 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:33:01.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:01 smithi028 ceph-mon[35823]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:33:01.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:01 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-06-18T02:33:01.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:01 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:33:02.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:02 smithi028 ceph-mon[35823]: Deploying daemon osd.0 on smithi001 2024-06-18T02:33:03.127 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:02 smithi001 ceph-mon[29099]: Deploying daemon osd.0 on smithi001 2024-06-18T02:33:03.871 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:03 smithi001 ceph-mon[29099]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:33:03.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:03 smithi028 ceph-mon[35823]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:33:05.914 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:05 smithi028 ceph-mon[35823]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:33:05.914 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:05 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:05.914 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:05 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:05.914 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:05 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi028", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-06-18T02:33:05.914 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:05 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:33:06.039 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:05 smithi001 ceph-mon[29099]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:33:06.039 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:05 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:06.039 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:05 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:06.039 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:05 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi028", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-06-18T02:33:06.039 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:05 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:33:06.953 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:06 smithi001 ceph-mon[29099]: Reconfiguring crash.smithi028 (monmap changed)... 2024-06-18T02:33:06.953 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:06 smithi001 ceph-mon[29099]: Reconfiguring daemon crash.smithi028 on smithi028 2024-06-18T02:33:06.953 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:06 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:33:06.953 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:06 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:06.953 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:06 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:06.953 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:06 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi028.xnuopx", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-18T02:33:06.953 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:06 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-18T02:33:06.954 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:06 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:33:06.984 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:06 smithi028 ceph-mon[35823]: Reconfiguring crash.smithi028 (monmap changed)... 2024-06-18T02:33:06.984 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:06 smithi028 ceph-mon[35823]: Reconfiguring daemon crash.smithi028 on smithi028 2024-06-18T02:33:06.984 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:06 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:33:06.984 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:06 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:06.984 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:06 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:06.984 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:06 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi028.xnuopx", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-18T02:33:06.984 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:06 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-18T02:33:06.984 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:06 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:33:07.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:07 smithi001 ceph-mon[29099]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:33:07.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:07 smithi001 ceph-mon[29099]: Reconfiguring mgr.smithi028.xnuopx (monmap changed)... 2024-06-18T02:33:07.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:07 smithi001 ceph-mon[29099]: Reconfiguring daemon mgr.smithi028.xnuopx on smithi028 2024-06-18T02:33:07.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:07 smithi028 ceph-mon[35823]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:33:07.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:07 smithi028 ceph-mon[35823]: Reconfiguring mgr.smithi028.xnuopx (monmap changed)... 2024-06-18T02:33:07.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:07 smithi028 ceph-mon[35823]: Reconfiguring daemon mgr.smithi028.xnuopx on smithi028 2024-06-18T02:33:09.013 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:08 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:09.013 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:08 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:09.013 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:08 smithi001 ceph-mon[29099]: Reconfiguring mon.smithi028 (monmap changed)... 2024-06-18T02:33:09.013 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:08 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-18T02:33:09.013 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:08 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-18T02:33:09.013 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:08 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:33:09.013 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:08 smithi001 ceph-mon[29099]: Reconfiguring daemon mon.smithi028 on smithi028 2024-06-18T02:33:09.094 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:08 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:09.094 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:08 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:09.094 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:08 smithi028 ceph-mon[35823]: Reconfiguring mon.smithi028 (monmap changed)... 2024-06-18T02:33:09.095 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:08 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-18T02:33:09.095 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:08 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-18T02:33:09.095 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:08 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:33:09.095 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:08 smithi028 ceph-mon[35823]: Reconfiguring daemon mon.smithi028 on smithi028 2024-06-18T02:33:10.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:09 smithi001 ceph-mon[29099]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:33:10.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:09 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:10.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:09 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:10.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:09 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-18T02:33:10.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:09 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi001.front.sepia.ceph.com:9093"}]: dispatch 2024-06-18T02:33:10.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:09 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:10.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:09 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-18T02:33:10.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:09 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi001.front.sepia.ceph.com:3000"}]: dispatch 2024-06-18T02:33:10.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:09 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:10.129 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:09 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-18T02:33:10.129 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:09 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi001.front.sepia.ceph.com:9095"}]: dispatch 2024-06-18T02:33:10.129 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:09 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:10.129 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:09 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:33:10.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:09 smithi028 ceph-mon[35823]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:33:10.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:09 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:10.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:09 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:10.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:09 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-18T02:33:10.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:09 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi001.front.sepia.ceph.com:9093"}]: dispatch 2024-06-18T02:33:10.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:09 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:10.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:09 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-18T02:33:10.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:09 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi001.front.sepia.ceph.com:3000"}]: dispatch 2024-06-18T02:33:10.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:09 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:10.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:09 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-18T02:33:10.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:09 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi001.front.sepia.ceph.com:9095"}]: dispatch 2024-06-18T02:33:10.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:09 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:10.239 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:09 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:33:11.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:10 smithi001 ceph-mon[29099]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-18T02:33:11.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:10 smithi001 ceph-mon[29099]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi001.front.sepia.ceph.com:9093"}]: dispatch 2024-06-18T02:33:11.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:10 smithi001 ceph-mon[29099]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-18T02:33:11.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:10 smithi001 ceph-mon[29099]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi001.front.sepia.ceph.com:3000"}]: dispatch 2024-06-18T02:33:11.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:10 smithi001 ceph-mon[29099]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-18T02:33:11.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:10 smithi001 ceph-mon[29099]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi001.front.sepia.ceph.com:9095"}]: dispatch 2024-06-18T02:33:11.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:10 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:11.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:10 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:11.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:10 smithi028 ceph-mon[35823]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-18T02:33:11.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:10 smithi028 ceph-mon[35823]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi001.front.sepia.ceph.com:9093"}]: dispatch 2024-06-18T02:33:11.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:10 smithi028 ceph-mon[35823]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-18T02:33:11.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:10 smithi028 ceph-mon[35823]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi001.front.sepia.ceph.com:3000"}]: dispatch 2024-06-18T02:33:11.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:10 smithi028 ceph-mon[35823]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-18T02:33:11.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:10 smithi028 ceph-mon[35823]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi001.front.sepia.ceph.com:9095"}]: dispatch 2024-06-18T02:33:11.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:10 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:11.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:10 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:12.176 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:11 smithi001 ceph-mon[29099]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:33:12.176 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:11 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:12.176 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:11 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:12.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:11 smithi028 ceph-mon[35823]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:33:12.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:11 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:12.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:11 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:13.118 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:12 smithi001 ceph-mon[29099]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:33:13.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:12 smithi028 ceph-mon[35823]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:33:14.095 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:13 smithi001 ceph-mon[29099]: from='osd.0 [v2:172.21.15.1:6802/797495043,v1:172.21.15.1:6803/797495043]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-06-18T02:33:14.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:14.129+0000 7f304f7fe640 1 -- 172.21.15.1:0/2533817626 <== mgr.14216 v2:172.21.15.1:6800/729353917 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f306810dc70 con 0x7f30500653a0 2024-06-18T02:33:14.130 INFO:teuthology.orchestra.run.smithi001.stdout:Created osd(s) 0 on host 'smithi001' 2024-06-18T02:33:14.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:14.131+0000 7f306d56a640 1 -- 172.21.15.1:0/2533817626 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f30500653a0 msgr2=0x7f3050067850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:33:14.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:14.131+0000 7f306d56a640 1 --2- 172.21.15.1:0/2533817626 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f30500653a0 0x7f3050067850 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3058004550 tx=0x7f3058009040 comp rx=0 tx=0).stop 2024-06-18T02:33:14.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:14.131+0000 7f306d56a640 1 -- 172.21.15.1:0/2533817626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30680fd0b0 msgr2=0x7f306810c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:33:14.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:14.131+0000 7f306d56a640 1 --2- 172.21.15.1:0/2533817626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30680fd0b0 0x7f306810c3a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f305c000c00 tx=0x7f305c009da0 comp rx=0 tx=0).stop 2024-06-18T02:33:14.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:14.131+0000 7f306d56a640 1 -- 172.21.15.1:0/2533817626 shutdown_connections 2024-06-18T02:33:14.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:14.131+0000 7f306d56a640 1 --2- 172.21.15.1:0/2533817626 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f30500653a0 0x7f3050067850 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:33:14.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:14.131+0000 7f306d56a640 1 --2- 172.21.15.1:0/2533817626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30680fd0b0 0x7f306810c3a0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:33:14.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:14.131+0000 7f306d56a640 1 --2- 172.21.15.1:0/2533817626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30680fc760 0x7f306810be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:33:14.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:14.131+0000 7f306d56a640 1 -- 172.21.15.1:0/2533817626 >> 172.21.15.1:0/2533817626 conn(0x7f30680f60f0 msgr2=0x7f30680f77d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:33:14.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:14.131+0000 7f306d56a640 1 -- 172.21.15.1:0/2533817626 shutdown_connections 2024-06-18T02:33:14.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:14.132+0000 7f306d56a640 1 -- 172.21.15.1:0/2533817626 wait complete. 2024-06-18T02:33:14.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:13 smithi028 ceph-mon[35823]: from='osd.0 [v2:172.21.15.1:6802/797495043,v1:172.21.15.1:6803/797495043]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-06-18T02:33:14.746 DEBUG:teuthology.orchestra.run.smithi001:osd.0> sudo journalctl -f -n 0 -u ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968@osd.0.service 2024-06-18T02:33:14.749 INFO:tasks.cephadm:Deploying osd.1 on smithi001 with /dev/vg_nvme/lv_3... 2024-06-18T02:33:14.749 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-06-18T02:33:15.004 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:14 smithi001 ceph-mon[29099]: from='osd.0 [v2:172.21.15.1:6802/797495043,v1:172.21.15.1:6803/797495043]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-06-18T02:33:15.004 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:14 smithi001 ceph-mon[29099]: osdmap e7: 1 total, 0 up, 1 in 2024-06-18T02:33:15.004 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:14 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-18T02:33:15.004 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:14 smithi001 ceph-mon[29099]: from='osd.0 [v2:172.21.15.1:6802/797495043,v1:172.21.15.1:6803/797495043]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]: dispatch 2024-06-18T02:33:15.004 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:14 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:15.004 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:14 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:15.004 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:14 smithi001 ceph-mon[29099]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:33:15.005 INFO:journalctl@ceph.osd.0.smithi001.stdout:Jun 18 02:33:14 smithi001 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-0[47116]: 2024-06-18T02:33:14.954+0000 7f47e6015640 -1 osd.0 0 waiting for initial osdmap 2024-06-18T02:33:15.005 INFO:journalctl@ceph.osd.0.smithi001.stdout:Jun 18 02:33:14 smithi001 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-0[47116]: 2024-06-18T02:33:14.959+0000 7f47dde24640 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-18T02:33:15.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:14 smithi028 ceph-mon[35823]: from='osd.0 [v2:172.21.15.1:6802/797495043,v1:172.21.15.1:6803/797495043]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-06-18T02:33:15.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:14 smithi028 ceph-mon[35823]: osdmap e7: 1 total, 0 up, 1 in 2024-06-18T02:33:15.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:14 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-18T02:33:15.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:14 smithi028 ceph-mon[35823]: from='osd.0 [v2:172.21.15.1:6802/797495043,v1:172.21.15.1:6803/797495043]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]: dispatch 2024-06-18T02:33:15.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:14 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:15.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:14 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:15.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:14 smithi028 ceph-mon[35823]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:33:15.397 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:33:16.127 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:15 smithi001 ceph-mon[29099]: from='osd.0 [v2:172.21.15.1:6802/797495043,v1:172.21.15.1:6803/797495043]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]': finished 2024-06-18T02:33:16.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:15 smithi001 ceph-mon[29099]: osdmap e8: 1 total, 0 up, 1 in 2024-06-18T02:33:16.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:15 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-18T02:33:16.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:15 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-18T02:33:16.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:15 smithi028 ceph-mon[35823]: from='osd.0 [v2:172.21.15.1:6802/797495043,v1:172.21.15.1:6803/797495043]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]': finished 2024-06-18T02:33:16.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:15 smithi028 ceph-mon[35823]: osdmap e8: 1 total, 0 up, 1 in 2024-06-18T02:33:16.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:15 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-18T02:33:16.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:15 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-18T02:33:17.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:16 smithi028 ceph-mon[35823]: purged_snaps scrub starts 2024-06-18T02:33:17.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:16 smithi028 ceph-mon[35823]: purged_snaps scrub ok 2024-06-18T02:33:17.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:16 smithi028 ceph-mon[35823]: osd.0 [v2:172.21.15.1:6802/797495043,v1:172.21.15.1:6803/797495043] boot 2024-06-18T02:33:17.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:16 smithi028 ceph-mon[35823]: osdmap e9: 1 total, 1 up, 1 in 2024-06-18T02:33:17.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:16 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-18T02:33:17.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:16 smithi028 ceph-mon[35823]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:33:17.357 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:16 smithi001 ceph-mon[29099]: purged_snaps scrub starts 2024-06-18T02:33:17.357 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:16 smithi001 ceph-mon[29099]: purged_snaps scrub ok 2024-06-18T02:33:17.357 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:16 smithi001 ceph-mon[29099]: osd.0 [v2:172.21.15.1:6802/797495043,v1:172.21.15.1:6803/797495043] boot 2024-06-18T02:33:17.357 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:16 smithi001 ceph-mon[29099]: osdmap e9: 1 total, 1 up, 1 in 2024-06-18T02:33:17.357 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:16 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-18T02:33:17.357 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:16 smithi001 ceph-mon[29099]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:33:17.676 INFO:teuthology.orchestra.run.smithi001.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-06-18T02:33:17.773 INFO:teuthology.orchestra.run.smithi001.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-06-18T02:33:17.785 INFO:teuthology.orchestra.run.smithi001.stderr: stderr: 10+0 records in 2024-06-18T02:33:17.785 INFO:teuthology.orchestra.run.smithi001.stderr:10+0 records out 2024-06-18T02:33:17.786 INFO:teuthology.orchestra.run.smithi001.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00981513 s, 1.1 GB/s 2024-06-18T02:33:17.786 INFO:teuthology.orchestra.run.smithi001.stderr:--> Zapping successful for: 2024-06-18T02:33:18.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:17 smithi028 ceph-mon[35823]: osdmap e10: 1 total, 1 up, 1 in 2024-06-18T02:33:18.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:17 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:18.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:17 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:18.327 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:17 smithi001 ceph-mon[29099]: osdmap e10: 1 total, 1 up, 1 in 2024-06-18T02:33:18.327 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:17 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:18.327 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:17 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:18.359 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph orch daemon add osd smithi001:vg_nvme/lv_3 2024-06-18T02:33:19.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:18 smithi028 ceph-mon[35823]: pgmap v34: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-18T02:33:19.336 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:18 smithi001 ceph-mon[29099]: pgmap v34: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-18T02:33:19.524 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:33:21.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:21.625+0000 7fb5d0ad5640 1 -- 172.21.15.1:0/98356059 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb5cc0fb1e0 msgr2=0x7fb5cc0fb5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:33:21.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:21.625+0000 7fb5d0ad5640 1 --2- 172.21.15.1:0/98356059 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb5cc0fb1e0 0x7fb5cc0fb5f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb5bc0048d0 tx=0x7fb5bc009da0 comp rx=0 tx=0).stop 2024-06-18T02:33:21.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:21.625+0000 7fb5d0ad5640 1 -- 172.21.15.1:0/98356059 shutdown_connections 2024-06-18T02:33:21.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:21.625+0000 7fb5d0ad5640 1 --2- 172.21.15.1:0/98356059 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5cc0fc4b0 0x7fb5cc0fc920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:33:21.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:21.625+0000 7fb5d0ad5640 1 --2- 172.21.15.1:0/98356059 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb5cc0fb1e0 0x7fb5cc0fb5f0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:33:21.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:21.625+0000 7fb5d0ad5640 1 -- 172.21.15.1:0/98356059 >> 172.21.15.1:0/98356059 conn(0x7fb5cc0f69a0 msgr2=0x7fb5cc0f8dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:33:21.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:21.625+0000 7fb5d0ad5640 1 -- 172.21.15.1:0/98356059 shutdown_connections 2024-06-18T02:33:21.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:21.625+0000 7fb5d0ad5640 1 -- 172.21.15.1:0/98356059 wait complete. 2024-06-18T02:33:21.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:21.626+0000 7fb5d0ad5640 1 Processor -- start 2024-06-18T02:33:21.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:21.626+0000 7fb5d0ad5640 1 -- start start 2024-06-18T02:33:21.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:21.626+0000 7fb5d0ad5640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb5cc0fb1e0 0x7fb5cc112000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:33:21.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:21.626+0000 7fb5d0ad5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5cc0fc4b0 0x7fb5cc114550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:33:21.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:21.626+0000 7fb5d0ad5640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5cc10dab0 con 0x7fb5cc0fb1e0 2024-06-18T02:33:21.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:21.626+0000 7fb5d0ad5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5cc066460 con 0x7fb5cc0fc4b0 2024-06-18T02:33:21.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:21.626+0000 7fb5ca59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb5cc0fb1e0 0x7fb5cc112000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:33:21.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:21.626+0000 7fb5ca59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb5cc0fb1e0 0x7fb5cc112000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44258/0 (socket says 172.21.15.1:44258) 2024-06-18T02:33:21.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:21.626+0000 7fb5ca59c640 1 -- 172.21.15.1:0/1825534619 learned_addr learned my addr 172.21.15.1:0/1825534619 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:33:21.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:21.626+0000 7fb5ca59c640 1 -- 172.21.15.1:0/1825534619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5cc0fc4b0 msgr2=0x7fb5cc114550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:33:21.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:21.626+0000 7fb5ca59c640 1 --2- 172.21.15.1:0/1825534619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5cc0fc4b0 0x7fb5cc114550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:33:21.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:21.626+0000 7fb5ca59c640 1 -- 172.21.15.1:0/1825534619 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5bc005040 con 0x7fb5cc0fb1e0 2024-06-18T02:33:21.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:21.627+0000 7fb5ca59c640 1 --2- 172.21.15.1:0/1825534619 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb5cc0fb1e0 0x7fb5cc112000 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb5bc003080 tx=0x7fb5bc009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:33:21.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:21.627+0000 7fb5c2ffd640 1 -- 172.21.15.1:0/1825534619 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb5bc038070 con 0x7fb5cc0fb1e0 2024-06-18T02:33:21.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:21.627+0000 7fb5c2ffd640 1 -- 172.21.15.1:0/1825534619 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb5bc032970 con 0x7fb5cc0fb1e0 2024-06-18T02:33:21.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:21.627+0000 7fb5d0ad5640 1 -- 172.21.15.1:0/1825534619 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5cc0666e0 con 0x7fb5cc0fb1e0 2024-06-18T02:33:21.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:21.627+0000 7fb5c2ffd640 1 -- 172.21.15.1:0/1825534619 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb5bc033820 con 0x7fb5cc0fb1e0 2024-06-18T02:33:21.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:21.627+0000 7fb5d0ad5640 1 -- 172.21.15.1:0/1825534619 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5cc114df0 con 0x7fb5cc0fb1e0 2024-06-18T02:33:21.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:21.628+0000 7fb5d0ad5640 1 -- 172.21.15.1:0/1825534619 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5cc0fa640 con 0x7fb5cc0fb1e0 2024-06-18T02:33:21.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:21.628+0000 7fb5c2ffd640 1 -- 172.21.15.1:0/1825534619 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 18) v1 ==== 83687+0+0 (secure 0 0 0) 0x7fb5bc030020 con 0x7fb5cc0fb1e0 2024-06-18T02:33:21.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:21.629+0000 7fb5c2ffd640 1 --2- 172.21.15.1:0/1825534619 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fb5b80653a0 0x7fb5b8067850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:33:21.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:21.629+0000 7fb5c2ffd640 1 -- 172.21.15.1:0/1825534619 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1978+0+0 (secure 0 0 0) 0x7fb5bc0a18a0 con 0x7fb5cc0fb1e0 2024-06-18T02:33:21.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:21.629+0000 7fb5c3fff640 1 --2- 172.21.15.1:0/1825534619 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fb5b80653a0 0x7fb5b8067850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:33:21.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:21.629+0000 7fb5c3fff640 1 --2- 172.21.15.1:0/1825534619 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fb5b80653a0 0x7fb5b8067850 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb5b4004af0 tx=0x7fb5b4007c30 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:33:21.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:21.633+0000 7fb5c2ffd640 1 -- 172.21.15.1:0/1825534619 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fb5bc070910 con 0x7fb5cc0fb1e0 2024-06-18T02:33:21.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:21.824+0000 7fb5d0ad5640 1 -- 172.21.15.1:0/1825534619 --> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi001:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fb5cc100820 con 0x7fb5b80653a0 2024-06-18T02:33:21.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:21 smithi001 ceph-mon[29099]: pgmap v35: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-18T02:33:21.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:21 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:33:21.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:21 smithi028 ceph-mon[35823]: pgmap v35: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-18T02:33:21.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:21 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:33:22.867 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:22 smithi028 ceph-mon[35823]: from='client.14262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi001:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-18T02:33:22.867 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:22 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-18T02:33:22.867 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:22 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-18T02:33:22.868 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:22 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:33:22.868 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:22 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:22.868 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:22 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:22.868 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:22 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-18T02:33:22.868 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:22 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:22.868 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:22 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:33:22.868 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:22 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T02:33:22.868 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:22 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:22.868 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:22 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:33:22.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:22 smithi001 ceph-mon[29099]: from='client.14262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi001:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-18T02:33:22.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:22 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-18T02:33:22.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:22 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-18T02:33:22.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:22 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:33:22.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:22 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:22.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:22 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:22.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:22 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-18T02:33:22.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:22 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:22.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:22 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:33:22.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:22 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T02:33:22.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:22 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:22.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:22 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:33:23.627 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:23 smithi001 ceph-mon[29099]: Detected new or changed devices on smithi001 2024-06-18T02:33:23.628 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:23 smithi001 ceph-mon[29099]: Adjusting osd_memory_target on smithi001 to 12868M 2024-06-18T02:33:23.628 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:23 smithi001 ceph-mon[29099]: pgmap v36: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-18T02:33:23.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:23 smithi028 ceph-mon[35823]: Detected new or changed devices on smithi001 2024-06-18T02:33:23.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:23 smithi028 ceph-mon[35823]: Adjusting osd_memory_target on smithi001 to 12868M 2024-06-18T02:33:23.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:23 smithi028 ceph-mon[35823]: pgmap v36: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-18T02:33:25.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:24 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:25.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:24 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:25.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:24 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/2498819551' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f9bb7834-4fa4-40b1-9b63-bd1a69a4e082"}]: dispatch 2024-06-18T02:33:25.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:24 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/2498819551' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f9bb7834-4fa4-40b1-9b63-bd1a69a4e082"}]': finished 2024-06-18T02:33:25.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:24 smithi001 ceph-mon[29099]: osdmap e11: 2 total, 1 up, 2 in 2024-06-18T02:33:25.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:24 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-18T02:33:25.129 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:24 smithi001 ceph-mon[29099]: pgmap v38: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-18T02:33:25.129 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:24 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/4272399905' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-18T02:33:25.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:24 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:25.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:24 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:25.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:24 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/2498819551' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f9bb7834-4fa4-40b1-9b63-bd1a69a4e082"}]: dispatch 2024-06-18T02:33:25.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:24 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/2498819551' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f9bb7834-4fa4-40b1-9b63-bd1a69a4e082"}]': finished 2024-06-18T02:33:25.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:24 smithi028 ceph-mon[35823]: osdmap e11: 2 total, 1 up, 2 in 2024-06-18T02:33:25.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:24 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-18T02:33:25.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:24 smithi028 ceph-mon[35823]: pgmap v38: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-18T02:33:25.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:24 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/4272399905' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-18T02:33:27.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:27 smithi001 ceph-mon[29099]: pgmap v39: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-18T02:33:27.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:27 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:27.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:27 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:27.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:27 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:33:27.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:27 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T02:33:27.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:27 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:27.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:27 smithi028 ceph-mon[35823]: pgmap v39: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-18T02:33:27.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:27 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:27.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:27 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:27.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:27 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:33:27.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:27 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T02:33:27.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:27 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:29.737 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:29 smithi028 ceph-mon[35823]: pgmap v40: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-18T02:33:29.873 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:29 smithi001 ceph-mon[29099]: pgmap v40: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-18T02:33:31.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:31 smithi001 ceph-mon[29099]: pgmap v41: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-18T02:33:31.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:31 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-06-18T02:33:31.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:31 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:33:31.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:31 smithi028 ceph-mon[35823]: pgmap v41: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-18T02:33:31.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:31 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-06-18T02:33:31.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:31 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:33:32.744 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:32 smithi001 ceph-mon[29099]: Deploying daemon osd.1 on smithi001 2024-06-18T02:33:32.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:32 smithi028 ceph-mon[35823]: Deploying daemon osd.1 on smithi001 2024-06-18T02:33:33.684 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:33 smithi001 ceph-mon[29099]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-18T02:33:33.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:33 smithi028 ceph-mon[35823]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-18T02:33:35.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:35 smithi001 ceph-mon[29099]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-18T02:33:35.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:35 smithi028 ceph-mon[35823]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-18T02:33:36.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:36 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:33:36.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:36 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:36.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:36 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:36.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:36 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:33:36.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:36 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:33:36.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:36 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:36.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:36 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:36.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:36 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:33:37.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:37 smithi001 ceph-mon[29099]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-18T02:33:37.944 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:37 smithi028 ceph-mon[35823]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-18T02:33:39.543 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:39 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:39.543 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:39 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:39.543 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:39 smithi001 ceph-mon[29099]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-18T02:33:39.738 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:39 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:39.738 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:39 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:39.738 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:39 smithi028 ceph-mon[35823]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-18T02:33:40.217 INFO:teuthology.orchestra.run.smithi001.stdout:Created osd(s) 1 on host 'smithi001' 2024-06-18T02:33:40.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:40.217+0000 7fb5c2ffd640 1 -- 172.21.15.1:0/1825534619 <== mgr.14216 v2:172.21.15.1:6800/729353917 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fb5cc100820 con 0x7fb5b80653a0 2024-06-18T02:33:40.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:40.219+0000 7fb5d0ad5640 1 -- 172.21.15.1:0/1825534619 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fb5b80653a0 msgr2=0x7fb5b8067850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:33:40.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:40.219+0000 7fb5d0ad5640 1 --2- 172.21.15.1:0/1825534619 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fb5b80653a0 0x7fb5b8067850 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb5b4004af0 tx=0x7fb5b4007c30 comp rx=0 tx=0).stop 2024-06-18T02:33:40.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:40.219+0000 7fb5d0ad5640 1 -- 172.21.15.1:0/1825534619 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb5cc0fb1e0 msgr2=0x7fb5cc112000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:33:40.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:40.219+0000 7fb5d0ad5640 1 --2- 172.21.15.1:0/1825534619 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb5cc0fb1e0 0x7fb5cc112000 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb5bc003080 tx=0x7fb5bc009da0 comp rx=0 tx=0).stop 2024-06-18T02:33:40.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:40.219+0000 7fb5d0ad5640 1 -- 172.21.15.1:0/1825534619 shutdown_connections 2024-06-18T02:33:40.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:40.219+0000 7fb5d0ad5640 1 --2- 172.21.15.1:0/1825534619 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fb5b80653a0 0x7fb5b8067850 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:33:40.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:40.219+0000 7fb5d0ad5640 1 --2- 172.21.15.1:0/1825534619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5cc0fc4b0 0x7fb5cc114550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:33:40.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:40.219+0000 7fb5d0ad5640 1 --2- 172.21.15.1:0/1825534619 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb5cc0fb1e0 0x7fb5cc112000 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:33:40.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:40.219+0000 7fb5d0ad5640 1 -- 172.21.15.1:0/1825534619 >> 172.21.15.1:0/1825534619 conn(0x7fb5cc0f69a0 msgr2=0x7fb5cc0f80a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:33:40.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:40.219+0000 7fb5d0ad5640 1 -- 172.21.15.1:0/1825534619 shutdown_connections 2024-06-18T02:33:40.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:40.219+0000 7fb5d0ad5640 1 -- 172.21.15.1:0/1825534619 wait complete. 2024-06-18T02:33:40.814 DEBUG:teuthology.orchestra.run.smithi001:osd.1> sudo journalctl -f -n 0 -u ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968@osd.1.service 2024-06-18T02:33:40.817 INFO:tasks.cephadm:Deploying osd.2 on smithi001 with /dev/vg_nvme/lv_2... 2024-06-18T02:33:40.817 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-06-18T02:33:41.487 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:41 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:41.487 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:41 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:41.488 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:41 smithi028 ceph-mon[35823]: from='osd.1 [v2:172.21.15.1:6810/2182416687,v1:172.21.15.1:6811/2182416687]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-06-18T02:33:41.488 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:41 smithi028 ceph-mon[35823]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-18T02:33:41.515 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:41 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:41.515 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:41 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:41.515 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:41 smithi001 ceph-mon[29099]: from='osd.1 [v2:172.21.15.1:6810/2182416687,v1:172.21.15.1:6811/2182416687]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-06-18T02:33:41.515 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:41 smithi001 ceph-mon[29099]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-18T02:33:41.746 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:33:42.487 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:42 smithi028 ceph-mon[35823]: from='osd.1 [v2:172.21.15.1:6810/2182416687,v1:172.21.15.1:6811/2182416687]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-06-18T02:33:42.488 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:42 smithi028 ceph-mon[35823]: osdmap e12: 2 total, 1 up, 2 in 2024-06-18T02:33:42.488 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:42 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-18T02:33:42.488 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:42 smithi028 ceph-mon[35823]: from='osd.1 [v2:172.21.15.1:6810/2182416687,v1:172.21.15.1:6811/2182416687]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]: dispatch 2024-06-18T02:33:42.627 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:42 smithi001 ceph-mon[29099]: from='osd.1 [v2:172.21.15.1:6810/2182416687,v1:172.21.15.1:6811/2182416687]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-06-18T02:33:42.627 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:42 smithi001 ceph-mon[29099]: osdmap e12: 2 total, 1 up, 2 in 2024-06-18T02:33:42.628 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:42 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-18T02:33:42.628 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:42 smithi001 ceph-mon[29099]: from='osd.1 [v2:172.21.15.1:6810/2182416687,v1:172.21.15.1:6811/2182416687]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]: dispatch 2024-06-18T02:33:42.628 INFO:journalctl@ceph.osd.1.smithi001.stdout:Jun 18 02:33:42 smithi001 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-1[54605]: 2024-06-18T02:33:42.224+0000 7f3123e76640 -1 osd.1 0 waiting for initial osdmap 2024-06-18T02:33:42.628 INFO:journalctl@ceph.osd.1.smithi001.stdout:Jun 18 02:33:42 smithi001 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-1[54605]: 2024-06-18T02:33:42.229+0000 7f311cc87640 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-18T02:33:43.485 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:43 smithi001 ceph-mon[29099]: from='osd.1 [v2:172.21.15.1:6810/2182416687,v1:172.21.15.1:6811/2182416687]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]': finished 2024-06-18T02:33:43.485 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:43 smithi001 ceph-mon[29099]: osdmap e13: 2 total, 1 up, 2 in 2024-06-18T02:33:43.485 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:43 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-18T02:33:43.485 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:43 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-18T02:33:43.485 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:43 smithi001 ceph-mon[29099]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-18T02:33:43.487 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:43 smithi028 ceph-mon[35823]: from='osd.1 [v2:172.21.15.1:6810/2182416687,v1:172.21.15.1:6811/2182416687]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]': finished 2024-06-18T02:33:43.487 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:43 smithi028 ceph-mon[35823]: osdmap e13: 2 total, 1 up, 2 in 2024-06-18T02:33:43.487 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:43 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-18T02:33:43.487 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:43 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-18T02:33:43.487 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:43 smithi028 ceph-mon[35823]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-18T02:33:44.184 INFO:teuthology.orchestra.run.smithi001.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-06-18T02:33:44.273 INFO:teuthology.orchestra.run.smithi001.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-06-18T02:33:44.285 INFO:teuthology.orchestra.run.smithi001.stderr: stderr: 10+0 records in 2024-06-18T02:33:44.285 INFO:teuthology.orchestra.run.smithi001.stderr:10+0 records out 2024-06-18T02:33:44.285 INFO:teuthology.orchestra.run.smithi001.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00988123 s, 1.1 GB/s 2024-06-18T02:33:44.285 INFO:teuthology.orchestra.run.smithi001.stderr:--> Zapping successful for: 2024-06-18T02:33:44.487 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:44 smithi028 ceph-mon[35823]: purged_snaps scrub starts 2024-06-18T02:33:44.488 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:44 smithi028 ceph-mon[35823]: purged_snaps scrub ok 2024-06-18T02:33:44.488 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:44 smithi028 ceph-mon[35823]: osd.1 [v2:172.21.15.1:6810/2182416687,v1:172.21.15.1:6811/2182416687] boot 2024-06-18T02:33:44.488 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:44 smithi028 ceph-mon[35823]: osdmap e14: 2 total, 2 up, 2 in 2024-06-18T02:33:44.488 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:44 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-18T02:33:44.510 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:44 smithi001 ceph-mon[29099]: purged_snaps scrub starts 2024-06-18T02:33:44.510 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:44 smithi001 ceph-mon[29099]: purged_snaps scrub ok 2024-06-18T02:33:44.510 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:44 smithi001 ceph-mon[29099]: osd.1 [v2:172.21.15.1:6810/2182416687,v1:172.21.15.1:6811/2182416687] boot 2024-06-18T02:33:44.510 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:44 smithi001 ceph-mon[29099]: osdmap e14: 2 total, 2 up, 2 in 2024-06-18T02:33:44.510 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:44 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-18T02:33:45.252 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph orch daemon add osd smithi001:vg_nvme/lv_2 2024-06-18T02:33:45.487 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:45 smithi028 ceph-mon[35823]: osdmap e15: 2 total, 2 up, 2 in 2024-06-18T02:33:45.487 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:45 smithi028 ceph-mon[35823]: pgmap v52: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-18T02:33:45.526 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:33:45.628 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:45 smithi001 ceph-mon[29099]: osdmap e15: 2 total, 2 up, 2 in 2024-06-18T02:33:45.628 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:45 smithi001 ceph-mon[29099]: pgmap v52: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-18T02:33:46.618 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:46 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:46.619 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:46 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:46.737 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:46 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:46.738 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:46 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:47.531 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:47 smithi001 ceph-mon[29099]: pgmap v53: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-18T02:33:47.737 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:47 smithi028 ceph-mon[35823]: pgmap v53: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-18T02:33:47.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:47.786+0000 7f38d807f640 1 -- 172.21.15.1:0/3437585089 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f38d00fc460 msgr2=0x7f38d00fc8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:33:47.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:47.786+0000 7f38d807f640 1 --2- 172.21.15.1:0/3437585089 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f38d00fc460 0x7f38d00fc8d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f38c00048d0 tx=0x7f38c0009da0 comp rx=0 tx=0).stop 2024-06-18T02:33:47.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:47.786+0000 7f38d807f640 1 -- 172.21.15.1:0/3437585089 shutdown_connections 2024-06-18T02:33:47.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:47.786+0000 7f38d807f640 1 --2- 172.21.15.1:0/3437585089 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f38d00fc460 0x7f38d00fc8d0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:33:47.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:47.786+0000 7f38d807f640 1 --2- 172.21.15.1:0/3437585089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38d00fb190 0x7f38d00fb5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:33:47.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:47.786+0000 7f38d807f640 1 -- 172.21.15.1:0/3437585089 >> 172.21.15.1:0/3437585089 conn(0x7f38d00f6970 msgr2=0x7f38d00f8d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:33:47.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:47.786+0000 7f38d807f640 1 -- 172.21.15.1:0/3437585089 shutdown_connections 2024-06-18T02:33:47.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:47.786+0000 7f38d807f640 1 -- 172.21.15.1:0/3437585089 wait complete. 2024-06-18T02:33:47.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:47.787+0000 7f38d807f640 1 Processor -- start 2024-06-18T02:33:47.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:47.787+0000 7f38d807f640 1 -- start start 2024-06-18T02:33:47.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:47.787+0000 7f38d807f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38d00fb190 0x7f38d00ff260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:33:47.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:47.787+0000 7f38d807f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f38d00fc460 0x7f38d00fd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:33:47.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:47.787+0000 7f38d807f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38d00fdce0 con 0x7f38d00fc460 2024-06-18T02:33:47.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:47.787+0000 7f38d807f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38d00fde20 con 0x7f38d00fb190 2024-06-18T02:33:47.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:47.787+0000 7f38d561a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f38d00fc460 0x7f38d00fd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:33:47.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:47.787+0000 7f38d561a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f38d00fc460 0x7f38d00fd7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58748/0 (socket says 172.21.15.1:58748) 2024-06-18T02:33:47.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:47.787+0000 7f38d561a640 1 -- 172.21.15.1:0/934216444 learned_addr learned my addr 172.21.15.1:0/934216444 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:33:47.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:47.787+0000 7f38d5e1b640 1 --2- 172.21.15.1:0/934216444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38d00fb190 0x7f38d00ff260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:33:47.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:47.787+0000 7f38d561a640 1 -- 172.21.15.1:0/934216444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38d00fb190 msgr2=0x7f38d00ff260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:33:47.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:47.787+0000 7f38d561a640 1 --2- 172.21.15.1:0/934216444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38d00fb190 0x7f38d00ff260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:33:47.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:47.787+0000 7f38d561a640 1 -- 172.21.15.1:0/934216444 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38c0005040 con 0x7f38d00fc460 2024-06-18T02:33:47.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:47.788+0000 7f38d561a640 1 --2- 172.21.15.1:0/934216444 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f38d00fc460 0x7f38d00fd7a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f38c0009f80 tx=0x7f38c0009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:33:47.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:47.788+0000 7f38c67fc640 1 -- 172.21.15.1:0/934216444 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f38c0038070 con 0x7f38d00fc460 2024-06-18T02:33:47.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:47.788+0000 7f38d807f640 1 -- 172.21.15.1:0/934216444 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38d00fe0a0 con 0x7f38d00fc460 2024-06-18T02:33:47.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:47.788+0000 7f38c67fc640 1 -- 172.21.15.1:0/934216444 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f38c0032970 con 0x7f38d00fc460 2024-06-18T02:33:47.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:47.788+0000 7f38c67fc640 1 -- 172.21.15.1:0/934216444 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f38c0033860 con 0x7f38d00fc460 2024-06-18T02:33:47.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:47.788+0000 7f38d807f640 1 -- 172.21.15.1:0/934216444 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38d00fe5e0 con 0x7f38d00fc460 2024-06-18T02:33:47.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:47.789+0000 7f38d807f640 1 -- 172.21.15.1:0/934216444 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38d00fa5f0 con 0x7f38d00fc460 2024-06-18T02:33:47.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:47.790+0000 7f38c67fc640 1 -- 172.21.15.1:0/934216444 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 18) v1 ==== 83687+0+0 (secure 0 0 0) 0x7f38c0030020 con 0x7f38d00fc460 2024-06-18T02:33:47.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:47.790+0000 7f38c67fc640 1 --2- 172.21.15.1:0/934216444 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f38bc0653a0 0x7f38bc067850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:33:47.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:47.790+0000 7f38c67fc640 1 -- 172.21.15.1:0/934216444 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 2410+0+0 (secure 0 0 0) 0x7f38c00a1970 con 0x7f38d00fc460 2024-06-18T02:33:47.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:47.790+0000 7f38d5e1b640 1 --2- 172.21.15.1:0/934216444 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f38bc0653a0 0x7f38bc067850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:33:47.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:47.791+0000 7f38d5e1b640 1 --2- 172.21.15.1:0/934216444 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f38bc0653a0 0x7f38bc067850 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f38d00ff010 tx=0x7f38cc00b040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:33:47.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:47.794+0000 7f38c67fc640 1 -- 172.21.15.1:0/934216444 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f38c0070860 con 0x7f38d00fc460 2024-06-18T02:33:47.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:33:47.981+0000 7f38d807f640 1 -- 172.21.15.1:0/934216444 --> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi001:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f38d0103e10 con 0x7f38bc0653a0 2024-06-18T02:33:48.434 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:48 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-18T02:33:48.434 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:48 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-18T02:33:48.434 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:48 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:33:48.639 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:48 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-18T02:33:48.639 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:48 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-18T02:33:48.639 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:48 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:33:49.596 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:49 smithi001 ceph-mon[29099]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi001:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-18T02:33:49.596 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:49 smithi001 ceph-mon[29099]: pgmap v54: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-18T02:33:49.737 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:49 smithi028 ceph-mon[35823]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi001:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-18T02:33:49.738 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:49 smithi028 ceph-mon[35823]: pgmap v54: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-18T02:33:51.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:50 smithi028 ceph-mon[35823]: Detected new or changed devices on smithi001 2024-06-18T02:33:51.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:50 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:51.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:50 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:51.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:50 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-18T02:33:51.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:50 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-18T02:33:51.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:50 smithi028 ceph-mon[35823]: Adjusting osd_memory_target on smithi001 to 6434M 2024-06-18T02:33:51.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:50 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:51.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:50 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:33:51.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:50 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T02:33:51.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:50 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:51.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:50 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:33:51.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:50 smithi028 ceph-mon[35823]: pgmap v55: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-18T02:33:51.265 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:50 smithi001 ceph-mon[29099]: Detected new or changed devices on smithi001 2024-06-18T02:33:51.265 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:50 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:51.265 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:50 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:51.265 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:50 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-18T02:33:51.265 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:50 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-18T02:33:51.265 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:50 smithi001 ceph-mon[29099]: Adjusting osd_memory_target on smithi001 to 6434M 2024-06-18T02:33:51.266 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:50 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:51.266 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:50 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:33:51.266 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:50 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T02:33:51.266 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:50 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:51.266 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:50 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:33:51.266 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:50 smithi001 ceph-mon[29099]: pgmap v55: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-18T02:33:52.236 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:51 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/4075543101' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "89223bec-bba5-4744-960e-5396f04ddba1"}]: dispatch 2024-06-18T02:33:52.237 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:51 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/4075543101' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "89223bec-bba5-4744-960e-5396f04ddba1"}]': finished 2024-06-18T02:33:52.237 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:51 smithi001 ceph-mon[29099]: osdmap e16: 3 total, 2 up, 3 in 2024-06-18T02:33:52.237 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:51 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-18T02:33:52.237 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:51 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:33:52.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:51 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/4075543101' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "89223bec-bba5-4744-960e-5396f04ddba1"}]: dispatch 2024-06-18T02:33:52.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:51 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/4075543101' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "89223bec-bba5-4744-960e-5396f04ddba1"}]': finished 2024-06-18T02:33:52.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:51 smithi028 ceph-mon[35823]: osdmap e16: 3 total, 2 up, 3 in 2024-06-18T02:33:52.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:51 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-18T02:33:52.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:51 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:33:53.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:52 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/1314522646' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-18T02:33:53.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:52 smithi028 ceph-mon[35823]: pgmap v57: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-18T02:33:53.342 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:52 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/1314522646' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-18T02:33:53.342 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:52 smithi001 ceph-mon[29099]: pgmap v57: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-18T02:33:55.650 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:55 smithi001 ceph-mon[29099]: pgmap v58: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-18T02:33:55.650 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:55 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:55.650 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:55 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:55.650 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:55 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:33:55.650 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:55 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T02:33:55.651 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:55 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:55.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:55 smithi028 ceph-mon[35823]: pgmap v58: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-18T02:33:55.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:55 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:55.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:55 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:55.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:55 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:33:55.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:55 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T02:33:55.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:55 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:33:57.594 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:57 smithi001 ceph-mon[29099]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-18T02:33:57.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:57 smithi028 ceph-mon[35823]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-18T02:33:58.860 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:58 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-06-18T02:33:58.861 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:58 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:33:58.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:58 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-06-18T02:33:58.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:58 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:33:59.737 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:59 smithi028 ceph-mon[35823]: Deploying daemon osd.2 on smithi001 2024-06-18T02:33:59.738 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:33:59 smithi028 ceph-mon[35823]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-18T02:33:59.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:59 smithi001 ceph-mon[29099]: Deploying daemon osd.2 on smithi001 2024-06-18T02:33:59.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:33:59 smithi001 ceph-mon[29099]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-18T02:34:01.817 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:01 smithi001 ceph-mon[29099]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-18T02:34:01.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:01 smithi028 ceph-mon[35823]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-18T02:34:03.831 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:03 smithi001 ceph-mon[29099]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-18T02:34:03.831 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:03 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:34:03.831 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:03 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:34:03.831 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:03 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:34:03.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:03 smithi028 ceph-mon[35823]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-18T02:34:03.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:03 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:34:03.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:03 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:34:03.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:03 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:34:05.748 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:05 smithi001 ceph-mon[29099]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-18T02:34:05.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:05 smithi028 ceph-mon[35823]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-18T02:34:06.732 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:06 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:34:06.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:06 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:34:07.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:34:07.037+0000 7f38c67fc640 1 -- 172.21.15.1:0/934216444 <== mgr.14216 v2:172.21.15.1:6800/729353917 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f38d0103e10 con 0x7f38bc0653a0 2024-06-18T02:34:07.037 INFO:teuthology.orchestra.run.smithi001.stdout:Created osd(s) 2 on host 'smithi001' 2024-06-18T02:34:07.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:34:07.039+0000 7f38d807f640 1 -- 172.21.15.1:0/934216444 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f38bc0653a0 msgr2=0x7f38bc067850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:34:07.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:34:07.039+0000 7f38d807f640 1 --2- 172.21.15.1:0/934216444 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f38bc0653a0 0x7f38bc067850 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f38d00ff010 tx=0x7f38cc00b040 comp rx=0 tx=0).stop 2024-06-18T02:34:07.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:34:07.039+0000 7f38d807f640 1 -- 172.21.15.1:0/934216444 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f38d00fc460 msgr2=0x7f38d00fd7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:34:07.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:34:07.039+0000 7f38d807f640 1 --2- 172.21.15.1:0/934216444 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f38d00fc460 0x7f38d00fd7a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f38c0009f80 tx=0x7f38c0009da0 comp rx=0 tx=0).stop 2024-06-18T02:34:07.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:34:07.039+0000 7f38d807f640 1 -- 172.21.15.1:0/934216444 shutdown_connections 2024-06-18T02:34:07.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:34:07.039+0000 7f38d807f640 1 --2- 172.21.15.1:0/934216444 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f38bc0653a0 0x7f38bc067850 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:34:07.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:34:07.039+0000 7f38d807f640 1 --2- 172.21.15.1:0/934216444 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f38d00fc460 0x7f38d00fd7a0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:34:07.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:34:07.039+0000 7f38d807f640 1 --2- 172.21.15.1:0/934216444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38d00fb190 0x7f38d00ff260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:34:07.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:34:07.039+0000 7f38d807f640 1 -- 172.21.15.1:0/934216444 >> 172.21.15.1:0/934216444 conn(0x7f38d00f6970 msgr2=0x7f38d00652f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:34:07.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:34:07.039+0000 7f38d807f640 1 -- 172.21.15.1:0/934216444 shutdown_connections 2024-06-18T02:34:07.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:34:07.039+0000 7f38d807f640 1 -- 172.21.15.1:0/934216444 wait complete. 2024-06-18T02:34:07.627 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:07 smithi001 ceph-mon[29099]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-18T02:34:07.628 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:07 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:34:07.628 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:07 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:34:07.628 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:07 smithi001 ceph-mon[29099]: from='osd.2 [v2:172.21.15.1:6818/3682942580,v1:172.21.15.1:6819/3682942580]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-18T02:34:07.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:07 smithi028 ceph-mon[35823]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-18T02:34:07.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:07 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:34:07.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:07 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:34:07.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:07 smithi028 ceph-mon[35823]: from='osd.2 [v2:172.21.15.1:6818/3682942580,v1:172.21.15.1:6819/3682942580]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-18T02:34:07.991 DEBUG:teuthology.orchestra.run.smithi001:osd.2> sudo journalctl -f -n 0 -u ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968@osd.2.service 2024-06-18T02:34:07.994 INFO:tasks.cephadm:Deploying osd.3 on smithi028 with /dev/vg_nvme/lv_4... 2024-06-18T02:34:07.994 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-06-18T02:34:08.269 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi028/config 2024-06-18T02:34:09.315 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:09 smithi028 ceph-mon[35823]: from='osd.2 [v2:172.21.15.1:6818/3682942580,v1:172.21.15.1:6819/3682942580]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-06-18T02:34:09.315 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:09 smithi028 ceph-mon[35823]: osdmap e17: 3 total, 2 up, 3 in 2024-06-18T02:34:09.315 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:09 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-18T02:34:09.315 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:09 smithi028 ceph-mon[35823]: from='osd.2 [v2:172.21.15.1:6818/3682942580,v1:172.21.15.1:6819/3682942580]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]: dispatch 2024-06-18T02:34:09.315 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:09 smithi028 ceph-mon[35823]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-18T02:34:09.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:09 smithi001 ceph-mon[29099]: from='osd.2 [v2:172.21.15.1:6818/3682942580,v1:172.21.15.1:6819/3682942580]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-06-18T02:34:09.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:09 smithi001 ceph-mon[29099]: osdmap e17: 3 total, 2 up, 3 in 2024-06-18T02:34:09.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:09 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-18T02:34:09.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:09 smithi001 ceph-mon[29099]: from='osd.2 [v2:172.21.15.1:6818/3682942580,v1:172.21.15.1:6819/3682942580]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]: dispatch 2024-06-18T02:34:09.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:09 smithi001 ceph-mon[29099]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-18T02:34:09.379 INFO:journalctl@ceph.osd.2.smithi001.stdout:Jun 18 02:34:09 smithi001 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-2[62339]: 2024-06-18T02:34:09.043+0000 7f664ac54640 -1 osd.2 0 waiting for initial osdmap 2024-06-18T02:34:09.379 INFO:journalctl@ceph.osd.2.smithi001.stdout:Jun 18 02:34:09 smithi001 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-2[62339]: 2024-06-18T02:34:09.048+0000 7f6643a87640 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-18T02:34:09.985 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-06-18T02:34:10.088 INFO:teuthology.orchestra.run.smithi028.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-06-18T02:34:10.099 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10+0 records in 2024-06-18T02:34:10.099 INFO:teuthology.orchestra.run.smithi028.stderr:10+0 records out 2024-06-18T02:34:10.099 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00872018 s, 1.2 GB/s 2024-06-18T02:34:10.099 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping successful for: 2024-06-18T02:34:10.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:10 smithi028 ceph-mon[35823]: from='osd.2 [v2:172.21.15.1:6818/3682942580,v1:172.21.15.1:6819/3682942580]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]': finished 2024-06-18T02:34:10.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:10 smithi028 ceph-mon[35823]: osdmap e18: 3 total, 2 up, 3 in 2024-06-18T02:34:10.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:10 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-18T02:34:10.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:10 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-18T02:34:10.377 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:10 smithi001 ceph-mon[29099]: from='osd.2 [v2:172.21.15.1:6818/3682942580,v1:172.21.15.1:6819/3682942580]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi001", "root=default"]}]': finished 2024-06-18T02:34:10.377 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:10 smithi001 ceph-mon[29099]: osdmap e18: 3 total, 2 up, 3 in 2024-06-18T02:34:10.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:10 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-18T02:34:10.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:10 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-18T02:34:10.804 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph orch daemon add osd smithi028:vg_nvme/lv_4 2024-06-18T02:34:11.066 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi028/config 2024-06-18T02:34:11.334 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:11 smithi028 ceph-mon[35823]: purged_snaps scrub starts 2024-06-18T02:34:11.334 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:11 smithi028 ceph-mon[35823]: purged_snaps scrub ok 2024-06-18T02:34:11.334 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:11 smithi028 ceph-mon[35823]: osd.2 [v2:172.21.15.1:6818/3682942580,v1:172.21.15.1:6819/3682942580] boot 2024-06-18T02:34:11.334 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:11 smithi028 ceph-mon[35823]: osdmap e19: 3 total, 3 up, 3 in 2024-06-18T02:34:11.334 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:11 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-18T02:34:11.334 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:11 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:34:11.334 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:11 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:34:11.334 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:11 smithi028 ceph-mon[35823]: pgmap v69: 0 pgs: ; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-06-18T02:34:11.334 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:11 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-06-18T02:34:11.377 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:11 smithi001 ceph-mon[29099]: purged_snaps scrub starts 2024-06-18T02:34:11.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:11 smithi001 ceph-mon[29099]: purged_snaps scrub ok 2024-06-18T02:34:11.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:11 smithi001 ceph-mon[29099]: osd.2 [v2:172.21.15.1:6818/3682942580,v1:172.21.15.1:6819/3682942580] boot 2024-06-18T02:34:11.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:11 smithi001 ceph-mon[29099]: osdmap e19: 3 total, 3 up, 3 in 2024-06-18T02:34:11.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:11 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-18T02:34:11.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:11 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:34:11.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:11 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:34:11.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:11 smithi001 ceph-mon[29099]: pgmap v69: 0 pgs: ; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-06-18T02:34:11.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:11 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-06-18T02:34:12.259 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:12 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-06-18T02:34:12.259 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:12 smithi028 ceph-mon[35823]: osdmap e20: 3 total, 3 up, 3 in 2024-06-18T02:34:12.259 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:12 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-06-18T02:34:12.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:12.462+0000 7f3f6356d640 1 -- 172.21.15.28:0/1040931755 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f5c0fc480 msgr2=0x7f3f5c0fc8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:34:12.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:12.462+0000 7f3f6356d640 1 --2- 172.21.15.28:0/1040931755 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f5c0fc480 0x7f3f5c0fc8f0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c0048d0 tx=0x7f3f4c009da0 comp rx=0 tx=0).stop 2024-06-18T02:34:12.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:12.462+0000 7f3f6356d640 1 -- 172.21.15.28:0/1040931755 shutdown_connections 2024-06-18T02:34:12.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:12.462+0000 7f3f6356d640 1 --2- 172.21.15.28:0/1040931755 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f5c0fc480 0x7f3f5c0fc8f0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:34:12.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:12.462+0000 7f3f6356d640 1 --2- 172.21.15.28:0/1040931755 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f5c0fb1b0 0x7f3f5c0fb5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:34:12.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:12.462+0000 7f3f6356d640 1 -- 172.21.15.28:0/1040931755 >> 172.21.15.28:0/1040931755 conn(0x7f3f5c0f6970 msgr2=0x7f3f5c0f8da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:34:12.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:12.462+0000 7f3f6356d640 1 -- 172.21.15.28:0/1040931755 shutdown_connections 2024-06-18T02:34:12.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:12.462+0000 7f3f6356d640 1 -- 172.21.15.28:0/1040931755 wait complete. 2024-06-18T02:34:12.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:12.462+0000 7f3f6356d640 1 Processor -- start 2024-06-18T02:34:12.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:12.463+0000 7f3f6356d640 1 -- start start 2024-06-18T02:34:12.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:12.463+0000 7f3f6356d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f5c0fb1b0 0x7f3f5c0ff2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:34:12.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:12.463+0000 7f3f6356d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f5c0fc480 0x7f3f5c0fd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:34:12.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:12.463+0000 7f3f6356d640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f5c0fdd50 con 0x7f3f5c0fc480 2024-06-18T02:34:12.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:12.463+0000 7f3f6356d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f5c0fde90 con 0x7f3f5c0fb1b0 2024-06-18T02:34:12.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:12.463+0000 7f3f61309640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f5c0fb1b0 0x7f3f5c0ff2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:34:12.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:12.463+0000 7f3f61309640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f5c0fb1b0 0x7f3f5c0ff2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60512/0 (socket says 172.21.15.28:60512) 2024-06-18T02:34:12.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:12.463+0000 7f3f61309640 1 -- 172.21.15.28:0/3406305188 learned_addr learned my addr 172.21.15.28:0/3406305188 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-06-18T02:34:12.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:12.463+0000 7f3f60b08640 1 --2- 172.21.15.28:0/3406305188 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f5c0fc480 0x7f3f5c0fd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:34:12.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:12.463+0000 7f3f61309640 1 -- 172.21.15.28:0/3406305188 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f5c0fc480 msgr2=0x7f3f5c0fd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:34:12.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:12.463+0000 7f3f61309640 1 --2- 172.21.15.28:0/3406305188 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f5c0fc480 0x7f3f5c0fd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:34:12.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:12.463+0000 7f3f61309640 1 -- 172.21.15.28:0/3406305188 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f4c005040 con 0x7f3f5c0fb1b0 2024-06-18T02:34:12.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:12.463+0000 7f3f61309640 1 --2- 172.21.15.28:0/3406305188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f5c0fb1b0 0x7f3f5c0ff2d0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f3f58002a50 tx=0x7f3f58004600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:34:12.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:12.463+0000 7f3f51ffb640 1 -- 172.21.15.28:0/3406305188 <== mon.1 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f58007c30 con 0x7f3f5c0fb1b0 2024-06-18T02:34:12.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:12.464+0000 7f3f51ffb640 1 -- 172.21.15.28:0/3406305188 <== mon.1 v2:172.21.15.28:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3f5800f040 con 0x7f3f5c0fb1b0 2024-06-18T02:34:12.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:12.464+0000 7f3f6356d640 1 -- 172.21.15.28:0/3406305188 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f5c0fe170 con 0x7f3f5c0fb1b0 2024-06-18T02:34:12.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:12.464+0000 7f3f51ffb640 1 -- 172.21.15.28:0/3406305188 <== mon.1 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f5800dad0 con 0x7f3f5c0fb1b0 2024-06-18T02:34:12.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:12.464+0000 7f3f6356d640 1 -- 172.21.15.28:0/3406305188 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f5c114f30 con 0x7f3f5c0fb1b0 2024-06-18T02:34:12.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:12.465+0000 7f3f51ffb640 1 -- 172.21.15.28:0/3406305188 <== mon.1 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) v1 ==== 83687+0+0 (secure 0 0 0) 0x7f3f5800dc70 con 0x7f3f5c0fb1b0 2024-06-18T02:34:12.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:12.465+0000 7f3f6356d640 1 -- 172.21.15.28:0/3406305188 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f5c0fa610 con 0x7f3f5c0fb1b0 2024-06-18T02:34:12.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:12.466+0000 7f3f51ffb640 1 --2- 172.21.15.28:0/3406305188 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f3f480653a0 0x7f3f48067850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:34:12.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:12.466+0000 7f3f51ffb640 1 -- 172.21.15.28:0/3406305188 <== mon.1 v2:172.21.15.28:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 3228+0+0 (secure 0 0 0) 0x7f3f58081570 con 0x7f3f5c0fb1b0 2024-06-18T02:34:12.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:12.466+0000 7f3f60b08640 1 --2- 172.21.15.28:0/3406305188 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f3f480653a0 0x7f3f48067850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:34:12.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:12.467+0000 7f3f60b08640 1 --2- 172.21.15.28:0/3406305188 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f3f480653a0 0x7f3f48067850 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c0043d0 tx=0x7f3f4c035040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:34:12.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:12.471+0000 7f3f51ffb640 1 -- 172.21.15.28:0/3406305188 <== mon.1 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f3f580501b0 con 0x7f3f5c0fb1b0 2024-06-18T02:34:12.628 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:12 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-06-18T02:34:12.628 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:12 smithi001 ceph-mon[29099]: osdmap e20: 3 total, 3 up, 3 in 2024-06-18T02:34:12.628 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:12 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-06-18T02:34:12.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:12.656+0000 7f3f6356d640 1 -- 172.21.15.28:0/3406305188 --> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f3f5c103e90 con 0x7f3f480653a0 2024-06-18T02:34:13.377 INFO:journalctl@ceph.osd.0.smithi001.stdout:Jun 18 02:34:13 smithi001 sudo[66879]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-18T02:34:13.377 INFO:journalctl@ceph.osd.0.smithi001.stdout:Jun 18 02:34:13 smithi001 sudo[66879]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-18T02:34:13.378 INFO:journalctl@ceph.osd.0.smithi001.stdout:Jun 18 02:34:13 smithi001 sudo[66879]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-18T02:34:13.378 INFO:journalctl@ceph.osd.0.smithi001.stdout:Jun 18 02:34:13 smithi001 sudo[66879]: pam_unix(sudo:session): session closed for user root 2024-06-18T02:34:13.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:13 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-06-18T02:34:13.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:13 smithi001 ceph-mon[29099]: osdmap e21: 3 total, 3 up, 3 in 2024-06-18T02:34:13.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:13 smithi001 ceph-mon[29099]: pgmap v72: 1 pgs: 1 unknown; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-06-18T02:34:13.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:13 smithi001 ceph-mon[29099]: from='client.24135 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-18T02:34:13.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:13 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-18T02:34:13.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:13 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-18T02:34:13.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:13 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:34:13.382 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:13 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-06-18T02:34:13.382 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:13 smithi028 ceph-mon[35823]: osdmap e21: 3 total, 3 up, 3 in 2024-06-18T02:34:13.382 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:13 smithi028 ceph-mon[35823]: pgmap v72: 1 pgs: 1 unknown; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-06-18T02:34:13.382 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:13 smithi028 ceph-mon[35823]: from='client.24135 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-18T02:34:13.382 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:13 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-18T02:34:13.382 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:13 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-18T02:34:13.383 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:13 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:34:13.877 INFO:journalctl@ceph.osd.0.smithi001.stdout:Jun 18 02:34:13 smithi001 sudo[66981]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-18T02:34:13.878 INFO:journalctl@ceph.osd.0.smithi001.stdout:Jun 18 02:34:13 smithi001 sudo[66981]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-18T02:34:13.878 INFO:journalctl@ceph.osd.0.smithi001.stdout:Jun 18 02:34:13 smithi001 sudo[66981]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-18T02:34:13.878 INFO:journalctl@ceph.osd.0.smithi001.stdout:Jun 18 02:34:13 smithi001 sudo[66981]: pam_unix(sudo:session): session closed for user root 2024-06-18T02:34:14.341 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:14 smithi028 ceph-mon[35823]: osdmap e22: 3 total, 3 up, 3 in 2024-06-18T02:34:14.341 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:14 smithi028 ceph-mon[35823]: Detected new or changed devices on smithi001 2024-06-18T02:34:14.341 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:14 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:34:14.341 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:14 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:34:14.341 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:14 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-18T02:34:14.341 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:14 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-18T02:34:14.342 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:14 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-18T02:34:14.342 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:14 smithi028 ceph-mon[35823]: Adjusting osd_memory_target on smithi001 to 4289M 2024-06-18T02:34:14.342 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:14 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:34:14.342 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:14 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:34:14.342 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:14 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T02:34:14.342 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:14 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:34:14.342 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:14 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:34:14.342 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:14 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:34:14.405 INFO:journalctl@ceph.osd.1.smithi001.stdout:Jun 18 02:34:14 smithi001 sudo[67068]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-18T02:34:14.405 INFO:journalctl@ceph.osd.1.smithi001.stdout:Jun 18 02:34:14 smithi001 sudo[67068]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-18T02:34:14.405 INFO:journalctl@ceph.osd.1.smithi001.stdout:Jun 18 02:34:14 smithi001 sudo[67068]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-18T02:34:14.405 INFO:journalctl@ceph.osd.1.smithi001.stdout:Jun 18 02:34:14 smithi001 sudo[67068]: pam_unix(sudo:session): session closed for user root 2024-06-18T02:34:14.405 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:14 smithi001 ceph-mon[29099]: osdmap e22: 3 total, 3 up, 3 in 2024-06-18T02:34:14.406 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:14 smithi001 ceph-mon[29099]: Detected new or changed devices on smithi001 2024-06-18T02:34:14.406 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:14 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:34:14.406 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:14 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:34:14.406 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:14 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-18T02:34:14.406 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:14 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-18T02:34:14.406 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:14 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-18T02:34:14.406 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:14 smithi001 ceph-mon[29099]: Adjusting osd_memory_target on smithi001 to 4289M 2024-06-18T02:34:14.406 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:14 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:34:14.406 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:14 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:34:14.406 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:14 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T02:34:14.406 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:14 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:34:14.406 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:14 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:34:14.406 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:14 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:34:14.877 INFO:journalctl@ceph.osd.1.smithi001.stdout:Jun 18 02:34:14 smithi001 sudo[67088]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-18T02:34:14.877 INFO:journalctl@ceph.osd.1.smithi001.stdout:Jun 18 02:34:14 smithi001 sudo[67088]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-18T02:34:14.877 INFO:journalctl@ceph.osd.1.smithi001.stdout:Jun 18 02:34:14 smithi001 sudo[67088]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-18T02:34:14.878 INFO:journalctl@ceph.osd.1.smithi001.stdout:Jun 18 02:34:14 smithi001 sudo[67088]: pam_unix(sudo:session): session closed for user root 2024-06-18T02:34:15.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:15.240+0000 7f3f51ffb640 1 -- 172.21.15.28:0/3406305188 <== mon.1 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7f3f58050580 con 0x7f3f5c0fb1b0 2024-06-18T02:34:15.377 INFO:journalctl@ceph.osd.2.smithi001.stdout:Jun 18 02:34:15 smithi001 sudo[67185]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-18T02:34:15.377 INFO:journalctl@ceph.osd.2.smithi001.stdout:Jun 18 02:34:15 smithi001 sudo[67185]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-18T02:34:15.378 INFO:journalctl@ceph.osd.2.smithi001.stdout:Jun 18 02:34:15 smithi001 sudo[67185]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-18T02:34:15.378 INFO:journalctl@ceph.osd.2.smithi001.stdout:Jun 18 02:34:15 smithi001 sudo[67185]: pam_unix(sudo:session): session closed for user root 2024-06-18T02:34:15.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:15 smithi001 ceph-mon[29099]: pgmap v74: 1 pgs: 1 active+clean; 577 KiB data, 872 MiB used, 267 GiB / 268 GiB avail 2024-06-18T02:34:15.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:15 smithi001 ceph-mon[29099]: from='client.? 172.21.15.28:0/2046664625' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "33dd638d-10d2-432a-92ac-74b68b4a8311"}]: dispatch 2024-06-18T02:34:15.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:15 smithi001 ceph-mon[29099]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "33dd638d-10d2-432a-92ac-74b68b4a8311"}]: dispatch 2024-06-18T02:34:15.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:15 smithi001 ceph-mon[29099]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "33dd638d-10d2-432a-92ac-74b68b4a8311"}]': finished 2024-06-18T02:34:15.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:15 smithi001 ceph-mon[29099]: osdmap e23: 4 total, 3 up, 4 in 2024-06-18T02:34:15.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:15 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-18T02:34:15.647 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:15 smithi028 ceph-mon[35823]: pgmap v74: 1 pgs: 1 active+clean; 577 KiB data, 872 MiB used, 267 GiB / 268 GiB avail 2024-06-18T02:34:15.647 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:15 smithi028 ceph-mon[35823]: from='client.? 172.21.15.28:0/2046664625' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "33dd638d-10d2-432a-92ac-74b68b4a8311"}]: dispatch 2024-06-18T02:34:15.648 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:15 smithi028 ceph-mon[35823]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "33dd638d-10d2-432a-92ac-74b68b4a8311"}]: dispatch 2024-06-18T02:34:15.648 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:15 smithi028 ceph-mon[35823]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "33dd638d-10d2-432a-92ac-74b68b4a8311"}]': finished 2024-06-18T02:34:15.648 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:15 smithi028 ceph-mon[35823]: osdmap e23: 4 total, 3 up, 4 in 2024-06-18T02:34:15.648 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:15 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-18T02:34:15.877 INFO:journalctl@ceph.osd.2.smithi001.stdout:Jun 18 02:34:15 smithi001 sudo[67201]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-18T02:34:15.877 INFO:journalctl@ceph.osd.2.smithi001.stdout:Jun 18 02:34:15 smithi001 sudo[67201]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-18T02:34:15.877 INFO:journalctl@ceph.osd.2.smithi001.stdout:Jun 18 02:34:15 smithi001 sudo[67201]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-18T02:34:15.878 INFO:journalctl@ceph.osd.2.smithi001.stdout:Jun 18 02:34:15 smithi001 sudo[67201]: pam_unix(sudo:session): session closed for user root 2024-06-18T02:34:16.377 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:15 smithi001 sudo[67269]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-18T02:34:16.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:15 smithi001 sudo[67269]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-18T02:34:16.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:15 smithi001 sudo[67269]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-18T02:34:16.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:15 smithi001 sudo[67269]: pam_unix(sudo:session): session closed for user root 2024-06-18T02:34:16.738 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:16 smithi028 ceph-mon[35823]: mgrmap e19: smithi001.teoawz(active, since 2m), standbys: smithi028.xnuopx 2024-06-18T02:34:16.739 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:16 smithi028 ceph-mon[35823]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-18T02:34:16.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:16 smithi001 sudo[67332]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-18T02:34:16.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:16 smithi001 sudo[67332]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-18T02:34:16.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:16 smithi001 sudo[67332]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-18T02:34:16.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:16 smithi001 sudo[67332]: pam_unix(sudo:session): session closed for user root 2024-06-18T02:34:16.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:16 smithi001 ceph-mon[29099]: mgrmap e19: smithi001.teoawz(active, since 2m), standbys: smithi028.xnuopx 2024-06-18T02:34:16.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:16 smithi001 ceph-mon[29099]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-18T02:34:17.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:16 smithi028 sudo[39556]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-18T02:34:17.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:16 smithi028 sudo[39556]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-18T02:34:17.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:16 smithi028 sudo[39556]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-18T02:34:17.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:16 smithi028 sudo[39556]: pam_unix(sudo:session): session closed for user root 2024-06-18T02:34:17.737 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:17 smithi028 sudo[39931]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-18T02:34:17.738 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:17 smithi028 sudo[39931]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-18T02:34:17.738 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:17 smithi028 sudo[39931]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-18T02:34:17.738 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:17 smithi028 sudo[39931]: pam_unix(sudo:session): session closed for user root 2024-06-18T02:34:17.738 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:17 smithi028 ceph-mon[35823]: from='client.? 172.21.15.28:0/320561167' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-18T02:34:17.738 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:17 smithi028 ceph-mon[35823]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-18T02:34:17.738 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:17 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "mon metadata", "id": "smithi001"}]: dispatch 2024-06-18T02:34:17.738 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:17 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "mon metadata", "id": "smithi028"}]: dispatch 2024-06-18T02:34:17.738 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:17 smithi028 ceph-mon[35823]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-18T02:34:17.738 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:17 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "mon metadata", "id": "smithi001"}]: dispatch 2024-06-18T02:34:17.738 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:17 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "mon metadata", "id": "smithi028"}]: dispatch 2024-06-18T02:34:17.739 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:17 smithi028 ceph-mon[35823]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-06-18T02:34:17.739 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:17 smithi028 ceph-mon[35823]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-18T02:34:17.739 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:17 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:34:17.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:17 smithi001 ceph-mon[29099]: from='client.? 172.21.15.28:0/320561167' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-18T02:34:17.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:17 smithi001 ceph-mon[29099]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-18T02:34:17.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:17 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "mon metadata", "id": "smithi001"}]: dispatch 2024-06-18T02:34:17.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:17 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "mon metadata", "id": "smithi028"}]: dispatch 2024-06-18T02:34:17.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:17 smithi001 ceph-mon[29099]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-18T02:34:17.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:17 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "mon metadata", "id": "smithi001"}]: dispatch 2024-06-18T02:34:17.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:17 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "mon metadata", "id": "smithi028"}]: dispatch 2024-06-18T02:34:17.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:17 smithi001 ceph-mon[29099]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-06-18T02:34:17.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:17 smithi001 ceph-mon[29099]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-18T02:34:17.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:17 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:34:18.943 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:18 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:34:18.943 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:18 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:34:18.943 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:18 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:34:18.943 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:18 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T02:34:18.944 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:18 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:34:19.070 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:18 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:34:19.070 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:18 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:34:19.070 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:18 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:34:19.070 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:18 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T02:34:19.070 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:18 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:34:19.963 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:19 smithi028 ceph-mon[35823]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-18T02:34:20.044 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:19 smithi001 ceph-mon[29099]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-18T02:34:21.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:21 smithi028 ceph-mon[35823]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-06-18T02:34:21.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:21 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:34:21.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:21 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-06-18T02:34:21.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:21 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:34:22.127 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:21 smithi001 ceph-mon[29099]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-06-18T02:34:22.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:21 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:34:22.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:21 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-06-18T02:34:22.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:21 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:34:23.127 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:22 smithi001 ceph-mon[29099]: Deploying daemon osd.3 on smithi028 2024-06-18T02:34:23.165 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:22 smithi028 ceph-mon[35823]: Deploying daemon osd.3 on smithi028 2024-06-18T02:34:23.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:23 smithi001 ceph-mon[29099]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-06-18T02:34:24.113 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:23 smithi028 ceph-mon[35823]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-06-18T02:34:25.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:25 smithi028 ceph-mon[35823]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-06-18T02:34:26.127 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:25 smithi001 ceph-mon[29099]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-06-18T02:34:27.015 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:26 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:34:27.015 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:26 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:34:27.015 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:26 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:34:27.127 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:26 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:34:27.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:26 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:34:27.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:26 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:34:27.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:27 smithi028 ceph-mon[35823]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.6 KiB/s rd, 68 KiB/s wr, 4 op/s 2024-06-18T02:34:28.127 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:27 smithi001 ceph-mon[29099]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.6 KiB/s rd, 68 KiB/s wr, 4 op/s 2024-06-18T02:34:29.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:29.274+0000 7f3f51ffb640 1 -- 172.21.15.28:0/3406305188 <== mgr.14216 v2:172.21.15.1:6800/729353917 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f3f5c103e90 con 0x7f3f480653a0 2024-06-18T02:34:29.274 INFO:teuthology.orchestra.run.smithi028.stdout:Created osd(s) 3 on host 'smithi028' 2024-06-18T02:34:29.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:29.276+0000 7f3f6356d640 1 -- 172.21.15.28:0/3406305188 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f3f480653a0 msgr2=0x7f3f48067850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:34:29.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:29.276+0000 7f3f6356d640 1 --2- 172.21.15.28:0/3406305188 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f3f480653a0 0x7f3f48067850 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c0043d0 tx=0x7f3f4c035040 comp rx=0 tx=0).stop 2024-06-18T02:34:29.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:29.276+0000 7f3f6356d640 1 -- 172.21.15.28:0/3406305188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f5c0fb1b0 msgr2=0x7f3f5c0ff2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:34:29.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:29.276+0000 7f3f6356d640 1 --2- 172.21.15.28:0/3406305188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f5c0fb1b0 0x7f3f5c0ff2d0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f3f58002a50 tx=0x7f3f58004600 comp rx=0 tx=0).stop 2024-06-18T02:34:29.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:29.276+0000 7f3f6356d640 1 -- 172.21.15.28:0/3406305188 shutdown_connections 2024-06-18T02:34:29.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:29.276+0000 7f3f6356d640 1 --2- 172.21.15.28:0/3406305188 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f3f480653a0 0x7f3f48067850 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:34:29.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:29.276+0000 7f3f6356d640 1 --2- 172.21.15.28:0/3406305188 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3f5c0fc480 0x7f3f5c0fd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:34:29.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:29.276+0000 7f3f6356d640 1 --2- 172.21.15.28:0/3406305188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f5c0fb1b0 0x7f3f5c0ff2d0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:34:29.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:29.276+0000 7f3f6356d640 1 -- 172.21.15.28:0/3406305188 >> 172.21.15.28:0/3406305188 conn(0x7f3f5c0f6970 msgr2=0x7f3f5c0f8050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:34:29.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:29.276+0000 7f3f6356d640 1 -- 172.21.15.28:0/3406305188 shutdown_connections 2024-06-18T02:34:29.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:29.276+0000 7f3f6356d640 1 -- 172.21.15.28:0/3406305188 wait complete. 2024-06-18T02:34:29.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:29 smithi001 ceph-mon[29099]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-06-18T02:34:29.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:29 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:34:29.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:29 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:34:29.934 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:29 smithi028 ceph-mon[35823]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-06-18T02:34:29.934 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:29 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:34:29.934 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:29 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:34:30.204 DEBUG:teuthology.orchestra.run.smithi028:osd.3> sudo journalctl -f -n 0 -u ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968@osd.3.service 2024-06-18T02:34:30.207 INFO:tasks.cephadm:Deploying osd.4 on smithi028 with /dev/vg_nvme/lv_3... 2024-06-18T02:34:30.207 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-06-18T02:34:30.737 INFO:journalctl@ceph.osd.3.smithi028.stdout:Jun 18 02:34:30 smithi028 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-3[41170]: 2024-06-18T02:34:30.463+0000 7f103edac2c0 -1 osd.3 0 log_to_monitors true 2024-06-18T02:34:30.842 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi028/config 2024-06-18T02:34:31.053 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:30 smithi028 ceph-mon[35823]: from='osd.3 [v2:172.21.15.28:6800/664150046,v1:172.21.15.28:6801/664150046]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-18T02:34:31.053 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:30 smithi028 ceph-mon[35823]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-18T02:34:31.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:30 smithi001 ceph-mon[29099]: from='osd.3 [v2:172.21.15.28:6800/664150046,v1:172.21.15.28:6801/664150046]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-18T02:34:31.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:30 smithi001 ceph-mon[29099]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-18T02:34:31.774 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:31 smithi028 ceph-mon[35823]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-06-18T02:34:31.774 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:31 smithi028 ceph-mon[35823]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-06-18T02:34:31.774 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:31 smithi028 ceph-mon[35823]: osdmap e24: 4 total, 3 up, 4 in 2024-06-18T02:34:31.774 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:31 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-18T02:34:31.774 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:31 smithi028 ceph-mon[35823]: from='osd.3 [v2:172.21.15.28:6800/664150046,v1:172.21.15.28:6801/664150046]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-06-18T02:34:31.774 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:31 smithi028 ceph-mon[35823]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-06-18T02:34:32.031 INFO:journalctl@ceph.osd.3.smithi028.stdout:Jun 18 02:34:31 smithi028 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-3[41170]: 2024-06-18T02:34:31.773+0000 7f103bd43640 -1 osd.3 0 waiting for initial osdmap 2024-06-18T02:34:32.031 INFO:journalctl@ceph.osd.3.smithi028.stdout:Jun 18 02:34:31 smithi028 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-3[41170]: 2024-06-18T02:34:31.782+0000 7f1035355640 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-18T02:34:32.127 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:31 smithi001 ceph-mon[29099]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-06-18T02:34:32.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:31 smithi001 ceph-mon[29099]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-06-18T02:34:32.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:31 smithi001 ceph-mon[29099]: osdmap e24: 4 total, 3 up, 4 in 2024-06-18T02:34:32.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:31 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-18T02:34:32.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:31 smithi001 ceph-mon[29099]: from='osd.3 [v2:172.21.15.28:6800/664150046,v1:172.21.15.28:6801/664150046]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-06-18T02:34:32.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:31 smithi001 ceph-mon[29099]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-06-18T02:34:32.504 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-06-18T02:34:32.602 INFO:teuthology.orchestra.run.smithi028.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-06-18T02:34:32.615 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10+0 records in 2024-06-18T02:34:32.615 INFO:teuthology.orchestra.run.smithi028.stderr:10+0 records out 2024-06-18T02:34:32.615 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114058 s, 919 MB/s 2024-06-18T02:34:32.615 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping successful for: 2024-06-18T02:34:33.069 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:32 smithi028 ceph-mon[35823]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-06-18T02:34:33.070 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:32 smithi028 ceph-mon[35823]: osdmap e25: 4 total, 3 up, 4 in 2024-06-18T02:34:33.070 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:32 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-18T02:34:33.070 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:32 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-18T02:34:33.070 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:32 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:34:33.070 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:32 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:34:33.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:32 smithi001 ceph-mon[29099]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-06-18T02:34:33.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:32 smithi001 ceph-mon[29099]: osdmap e25: 4 total, 3 up, 4 in 2024-06-18T02:34:33.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:32 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-18T02:34:33.129 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:32 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-18T02:34:33.129 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:32 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:34:33.129 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:32 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:34:33.878 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:33 smithi028 ceph-mon[35823]: purged_snaps scrub starts 2024-06-18T02:34:33.878 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:33 smithi028 ceph-mon[35823]: purged_snaps scrub ok 2024-06-18T02:34:33.878 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:33 smithi028 ceph-mon[35823]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-18T02:34:33.878 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:33 smithi028 ceph-mon[35823]: osd.3 [v2:172.21.15.28:6800/664150046,v1:172.21.15.28:6801/664150046] boot 2024-06-18T02:34:33.878 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:33 smithi028 ceph-mon[35823]: osdmap e26: 4 total, 4 up, 4 in 2024-06-18T02:34:33.878 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:33 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-18T02:34:33.878 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:33 smithi028 ceph-mon[35823]: osdmap e27: 4 total, 4 up, 4 in 2024-06-18T02:34:33.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:33 smithi001 ceph-mon[29099]: purged_snaps scrub starts 2024-06-18T02:34:33.879 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:33 smithi001 ceph-mon[29099]: purged_snaps scrub ok 2024-06-18T02:34:33.879 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:33 smithi001 ceph-mon[29099]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-18T02:34:33.879 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:33 smithi001 ceph-mon[29099]: osd.3 [v2:172.21.15.28:6800/664150046,v1:172.21.15.28:6801/664150046] boot 2024-06-18T02:34:33.879 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:33 smithi001 ceph-mon[29099]: osdmap e26: 4 total, 4 up, 4 in 2024-06-18T02:34:33.879 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:33 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-18T02:34:33.879 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:33 smithi001 ceph-mon[29099]: osdmap e27: 4 total, 4 up, 4 in 2024-06-18T02:34:34.554 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph orch daemon add osd smithi028:vg_nvme/lv_3 2024-06-18T02:34:35.353 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi028/config 2024-06-18T02:34:35.377 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:35 smithi001 ceph-mon[29099]: osdmap e28: 4 total, 4 up, 4 in 2024-06-18T02:34:35.377 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:35 smithi001 ceph-mon[29099]: pgmap v90: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 153 KiB/s, 0 objects/s recovering 2024-06-18T02:34:35.476 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:35 smithi028 ceph-mon[35823]: osdmap e28: 4 total, 4 up, 4 in 2024-06-18T02:34:35.476 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:35 smithi028 ceph-mon[35823]: pgmap v90: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 153 KiB/s, 0 objects/s recovering 2024-06-18T02:34:36.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:36 smithi001 ceph-mon[29099]: osdmap e29: 4 total, 4 up, 4 in 2024-06-18T02:34:36.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:36 smithi001 ceph-mon[29099]: Detected new or changed devices on smithi028 2024-06-18T02:34:36.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:36 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:34:36.379 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:36 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:34:36.379 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:36 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-18T02:34:36.379 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:36 smithi001 ceph-mon[29099]: Adjusting osd_memory_target on smithi028 to 15940M 2024-06-18T02:34:36.379 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:36 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:34:36.379 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:36 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:34:36.379 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:36 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T02:34:36.380 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:36 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:34:36.380 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:36 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:34:36.380 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:36 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:34:36.380 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:36 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T02:34:36.380 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:36 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:34:36.488 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:36 smithi028 ceph-mon[35823]: osdmap e29: 4 total, 4 up, 4 in 2024-06-18T02:34:36.488 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:36 smithi028 ceph-mon[35823]: Detected new or changed devices on smithi028 2024-06-18T02:34:36.488 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:36 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:34:36.488 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:36 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:34:36.488 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:36 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-18T02:34:36.489 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:36 smithi028 ceph-mon[35823]: Adjusting osd_memory_target on smithi028 to 15940M 2024-06-18T02:34:36.489 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:36 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:34:36.489 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:36 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:34:36.489 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:36 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T02:34:36.489 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:36 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:34:36.489 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:36 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:34:36.489 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:36 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:34:36.489 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:36 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T02:34:36.489 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:36 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:34:37.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:37.009+0000 7f63a87b4640 1 -- 172.21.15.28:0/770919390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63a00fc2c0 msgr2=0x7f63a00fc6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:34:37.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:37.009+0000 7f63a87b4640 1 --2- 172.21.15.28:0/770919390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63a00fc2c0 0x7f63a00fc6d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f639c0048d0 tx=0x7f639c009da0 comp rx=0 tx=0).stop 2024-06-18T02:34:37.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:37.009+0000 7f63a87b4640 1 -- 172.21.15.28:0/770919390 shutdown_connections 2024-06-18T02:34:37.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:37.009+0000 7f63a87b4640 1 --2- 172.21.15.28:0/770919390 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63a00fd590 0x7f63a00fda00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:34:37.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:37.009+0000 7f63a87b4640 1 --2- 172.21.15.28:0/770919390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63a00fc2c0 0x7f63a00fc6d0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:34:37.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:37.009+0000 7f63a87b4640 1 -- 172.21.15.28:0/770919390 >> 172.21.15.28:0/770919390 conn(0x7f63a00f7aa0 msgr2=0x7f63a00f9eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:34:37.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:37.009+0000 7f63a87b4640 1 -- 172.21.15.28:0/770919390 shutdown_connections 2024-06-18T02:34:37.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:37.009+0000 7f63a87b4640 1 -- 172.21.15.28:0/770919390 wait complete. 2024-06-18T02:34:37.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:37.010+0000 7f63a87b4640 1 Processor -- start 2024-06-18T02:34:37.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:37.010+0000 7f63a87b4640 1 -- start start 2024-06-18T02:34:37.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:37.010+0000 7f63a87b4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63a00fc2c0 0x7f63a010f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:34:37.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:37.010+0000 7f63a87b4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63a00fd590 0x7f63a010f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:34:37.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:37.010+0000 7f63a87b4640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63a010fc60 con 0x7f63a00fd590 2024-06-18T02:34:37.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:37.010+0000 7f63a87b4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63a0110480 con 0x7f63a00fc2c0 2024-06-18T02:34:37.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:37.010+0000 7f63a6550640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63a00fc2c0 0x7f63a010f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:34:37.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:37.010+0000 7f63a6550640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63a00fc2c0 0x7f63a010f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58752/0 (socket says 172.21.15.28:58752) 2024-06-18T02:34:37.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:37.010+0000 7f63a6550640 1 -- 172.21.15.28:0/3462259466 learned_addr learned my addr 172.21.15.28:0/3462259466 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-06-18T02:34:37.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:37.010+0000 7f63a5d4f640 1 --2- 172.21.15.28:0/3462259466 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63a00fd590 0x7f63a010f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:34:37.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:37.010+0000 7f63a6550640 1 -- 172.21.15.28:0/3462259466 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63a00fd590 msgr2=0x7f63a010f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:34:37.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:37.010+0000 7f63a6550640 1 --2- 172.21.15.28:0/3462259466 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63a00fd590 0x7f63a010f610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:34:37.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:37.011+0000 7f63a6550640 1 -- 172.21.15.28:0/3462259466 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f639c005040 con 0x7f63a00fc2c0 2024-06-18T02:34:37.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:37.011+0000 7f63a6550640 1 --2- 172.21.15.28:0/3462259466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63a00fc2c0 0x7f63a010f0d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f639c009f80 tx=0x7f639c009da0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:34:37.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:37.011+0000 7f6396ffd640 1 -- 172.21.15.28:0/3462259466 <== mon.1 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f639c038070 con 0x7f63a00fc2c0 2024-06-18T02:34:37.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:37.011+0000 7f63a87b4640 1 -- 172.21.15.28:0/3462259466 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63a01106b0 con 0x7f63a00fc2c0 2024-06-18T02:34:37.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:37.011+0000 7f6396ffd640 1 -- 172.21.15.28:0/3462259466 <== mon.1 v2:172.21.15.28:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f639c032970 con 0x7f63a00fc2c0 2024-06-18T02:34:37.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:37.011+0000 7f6396ffd640 1 -- 172.21.15.28:0/3462259466 <== mon.1 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f639c033860 con 0x7f63a00fc2c0 2024-06-18T02:34:37.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:37.011+0000 7f63a87b4640 1 -- 172.21.15.28:0/3462259466 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63a0110b70 con 0x7f63a00fc2c0 2024-06-18T02:34:37.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:37.012+0000 7f63a87b4640 1 -- 172.21.15.28:0/3462259466 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63a00fb720 con 0x7f63a00fc2c0 2024-06-18T02:34:37.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:37.012+0000 7f6396ffd640 1 -- 172.21.15.28:0/3462259466 <== mon.1 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7f639c030020 con 0x7f63a00fc2c0 2024-06-18T02:34:37.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:37.013+0000 7f6396ffd640 1 --2- 172.21.15.28:0/3462259466 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f638c065430 0x7f638c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:34:37.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:37.013+0000 7f6396ffd640 1 -- 172.21.15.28:0/3462259466 <== mon.1 v2:172.21.15.28:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 3770+0+0 (secure 0 0 0) 0x7f639c0a2080 con 0x7f63a00fc2c0 2024-06-18T02:34:37.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:37.013+0000 7f63a5d4f640 1 --2- 172.21.15.28:0/3462259466 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f638c065430 0x7f638c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:34:37.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:37.013+0000 7f63a5d4f640 1 --2- 172.21.15.28:0/3462259466 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f638c065430 0x7f638c0678e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6390009af0 tx=0x7f639000b040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:34:37.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:37.017+0000 7f6396ffd640 1 -- 172.21.15.28:0/3462259466 <== mon.1 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f639c070890 con 0x7f63a00fc2c0 2024-06-18T02:34:37.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:37.198+0000 7f63a87b4640 1 -- 172.21.15.28:0/3462259466 --> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f63a0104fc0 con 0x7f638c065430 2024-06-18T02:34:37.198 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:37 smithi028 ceph-mon[35823]: osdmap e30: 4 total, 4 up, 4 in 2024-06-18T02:34:37.198 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:37 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:34:37.198 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:37 smithi028 ceph-mon[35823]: pgmap v93: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 153 KiB/s, 0 objects/s recovering 2024-06-18T02:34:37.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:37 smithi001 ceph-mon[29099]: osdmap e30: 4 total, 4 up, 4 in 2024-06-18T02:34:37.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:37 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:34:37.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:37 smithi001 ceph-mon[29099]: pgmap v93: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 153 KiB/s, 0 objects/s recovering 2024-06-18T02:34:38.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:38 smithi028 ceph-mon[35823]: from='client.24159 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-18T02:34:38.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:38 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-18T02:34:38.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:38 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-18T02:34:38.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:38 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:34:38.377 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:38 smithi001 ceph-mon[29099]: from='client.24159 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-18T02:34:38.377 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:38 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-18T02:34:38.377 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:38 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-18T02:34:38.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:38 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:34:39.206 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:39 smithi028 ceph-mon[35823]: pgmap v94: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-06-18T02:34:39.377 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:39 smithi001 ceph-mon[29099]: pgmap v94: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-06-18T02:34:40.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:40 smithi001 ceph-mon[29099]: from='client.? 172.21.15.28:0/738062412' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2f4a6526-2f3e-417f-b02c-cf242e7eac07"}]: dispatch 2024-06-18T02:34:40.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:40 smithi001 ceph-mon[29099]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2f4a6526-2f3e-417f-b02c-cf242e7eac07"}]: dispatch 2024-06-18T02:34:40.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:40 smithi001 ceph-mon[29099]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2f4a6526-2f3e-417f-b02c-cf242e7eac07"}]': finished 2024-06-18T02:34:40.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:40 smithi001 ceph-mon[29099]: osdmap e31: 5 total, 4 up, 5 in 2024-06-18T02:34:40.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:40 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-18T02:34:40.487 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:40 smithi028 ceph-mon[35823]: from='client.? 172.21.15.28:0/738062412' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2f4a6526-2f3e-417f-b02c-cf242e7eac07"}]: dispatch 2024-06-18T02:34:40.488 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:40 smithi028 ceph-mon[35823]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2f4a6526-2f3e-417f-b02c-cf242e7eac07"}]: dispatch 2024-06-18T02:34:40.488 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:40 smithi028 ceph-mon[35823]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2f4a6526-2f3e-417f-b02c-cf242e7eac07"}]': finished 2024-06-18T02:34:40.488 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:40 smithi028 ceph-mon[35823]: osdmap e31: 5 total, 4 up, 5 in 2024-06-18T02:34:40.488 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:40 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-18T02:34:41.377 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:41 smithi001 ceph-mon[29099]: from='client.? 172.21.15.28:0/3698205985' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-18T02:34:41.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:41 smithi001 ceph-mon[29099]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-18T02:34:41.487 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:41 smithi028 ceph-mon[35823]: from='client.? 172.21.15.28:0/3698205985' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-18T02:34:41.487 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:41 smithi028 ceph-mon[35823]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-18T02:34:43.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:43 smithi001 ceph-mon[29099]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-18T02:34:43.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:43 smithi028 ceph-mon[35823]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-18T02:34:45.737 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:45 smithi028 ceph-mon[35823]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-18T02:34:45.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:45 smithi001 ceph-mon[29099]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-18T02:34:46.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:46 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-06-18T02:34:46.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:46 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:34:46.896 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:46 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-06-18T02:34:46.896 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:46 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:34:47.836 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:47 smithi028 ceph-mon[35823]: Deploying daemon osd.4 on smithi028 2024-06-18T02:34:47.837 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:47 smithi028 ceph-mon[35823]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-18T02:34:47.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:47 smithi001 ceph-mon[29099]: Deploying daemon osd.4 on smithi028 2024-06-18T02:34:47.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:47 smithi001 ceph-mon[29099]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-18T02:34:49.738 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:49 smithi028 ceph-mon[35823]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-18T02:34:49.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:49 smithi001 ceph-mon[29099]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-18T02:34:51.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:51 smithi001 ceph-mon[29099]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-18T02:34:51.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:51 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:34:51.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:51 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:34:51.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:51 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:34:51.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:51 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:34:51.921 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:51 smithi028 ceph-mon[35823]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-18T02:34:51.922 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:51 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:34:51.922 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:51 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:34:51.922 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:51 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:34:51.922 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:51 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:34:53.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:53 smithi001 ceph-mon[29099]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-18T02:34:53.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:53.885+0000 7f6396ffd640 1 -- 172.21.15.28:0/3462259466 <== mgr.14216 v2:172.21.15.1:6800/729353917 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f639000f040 con 0x7f638c065430 2024-06-18T02:34:53.885 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:53 smithi028 ceph-mon[35823]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-18T02:34:53.885 INFO:teuthology.orchestra.run.smithi028.stdout:Created osd(s) 4 on host 'smithi028' 2024-06-18T02:34:53.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:53.887+0000 7f63a87b4640 1 -- 172.21.15.28:0/3462259466 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f638c065430 msgr2=0x7f638c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:34:53.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:53.887+0000 7f63a87b4640 1 --2- 172.21.15.28:0/3462259466 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f638c065430 0x7f638c0678e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6390009af0 tx=0x7f639000b040 comp rx=0 tx=0).stop 2024-06-18T02:34:53.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:53.887+0000 7f63a87b4640 1 -- 172.21.15.28:0/3462259466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63a00fc2c0 msgr2=0x7f63a010f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:34:53.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:53.887+0000 7f63a87b4640 1 --2- 172.21.15.28:0/3462259466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63a00fc2c0 0x7f63a010f0d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f639c009f80 tx=0x7f639c009da0 comp rx=0 tx=0).stop 2024-06-18T02:34:53.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:53.887+0000 7f63a87b4640 1 -- 172.21.15.28:0/3462259466 shutdown_connections 2024-06-18T02:34:53.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:53.887+0000 7f63a87b4640 1 --2- 172.21.15.28:0/3462259466 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f638c065430 0x7f638c0678e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:34:53.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:53.887+0000 7f63a87b4640 1 --2- 172.21.15.28:0/3462259466 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63a00fd590 0x7f63a010f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:34:53.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:53.887+0000 7f63a87b4640 1 --2- 172.21.15.28:0/3462259466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63a00fc2c0 0x7f63a010f0d0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:34:53.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:53.887+0000 7f63a87b4640 1 -- 172.21.15.28:0/3462259466 >> 172.21.15.28:0/3462259466 conn(0x7f63a00f7aa0 msgr2=0x7f63a00f9180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:34:53.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:53.888+0000 7f63a87b4640 1 -- 172.21.15.28:0/3462259466 shutdown_connections 2024-06-18T02:34:53.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:34:53.888+0000 7f63a87b4640 1 -- 172.21.15.28:0/3462259466 wait complete. 2024-06-18T02:34:54.810 DEBUG:teuthology.orchestra.run.smithi028:osd.4> sudo journalctl -f -n 0 -u ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968@osd.4.service 2024-06-18T02:34:54.812 INFO:tasks.cephadm:Deploying osd.5 on smithi028 with /dev/vg_nvme/lv_2... 2024-06-18T02:34:54.813 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-06-18T02:34:55.081 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:54 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:34:55.081 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:54 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:34:55.377 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:54 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:34:55.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:54 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:34:55.487 INFO:journalctl@ceph.osd.4.smithi028.stdout:Jun 18 02:34:55 smithi028 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-4[47983]: 2024-06-18T02:34:55.081+0000 7f864df542c0 -1 osd.4 0 log_to_monitors true 2024-06-18T02:34:55.534 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi028/config 2024-06-18T02:34:56.180 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:55 smithi028 ceph-mon[35823]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-18T02:34:56.180 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:55 smithi028 ceph-mon[35823]: from='osd.4 [v2:172.21.15.28:6808/276884153,v1:172.21.15.28:6809/276884153]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-18T02:34:56.180 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:55 smithi028 ceph-mon[35823]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-18T02:34:56.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:55 smithi001 ceph-mon[29099]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-18T02:34:56.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:55 smithi001 ceph-mon[29099]: from='osd.4 [v2:172.21.15.28:6808/276884153,v1:172.21.15.28:6809/276884153]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-18T02:34:56.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:55 smithi001 ceph-mon[29099]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-18T02:34:57.193 INFO:journalctl@ceph.osd.4.smithi028.stdout:Jun 18 02:34:56 smithi028 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-4[47983]: 2024-06-18T02:34:56.896+0000 7f86496c6640 -1 osd.4 0 waiting for initial osdmap 2024-06-18T02:34:57.193 INFO:journalctl@ceph.osd.4.smithi028.stdout:Jun 18 02:34:56 smithi028 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-4[47983]: 2024-06-18T02:34:56.903+0000 7f8645d00640 -1 osd.4 33 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-18T02:34:57.194 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:56 smithi028 ceph-mon[35823]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-06-18T02:34:57.194 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:56 smithi028 ceph-mon[35823]: osdmap e32: 5 total, 4 up, 5 in 2024-06-18T02:34:57.194 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:56 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-18T02:34:57.194 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:56 smithi028 ceph-mon[35823]: from='osd.4 [v2:172.21.15.28:6808/276884153,v1:172.21.15.28:6809/276884153]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-06-18T02:34:57.194 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:56 smithi028 ceph-mon[35823]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-06-18T02:34:57.194 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:56 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:34:57.194 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:56 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:34:57.230 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-06-18T02:34:57.346 INFO:teuthology.orchestra.run.smithi028.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-06-18T02:34:57.357 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10+0 records in 2024-06-18T02:34:57.357 INFO:teuthology.orchestra.run.smithi028.stderr:10+0 records out 2024-06-18T02:34:57.357 INFO:teuthology.orchestra.run.smithi028.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00984683 s, 1.1 GB/s 2024-06-18T02:34:57.357 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 2024-06-18T02:34:57.357 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping successful for: 2024-06-18T02:34:57.377 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:56 smithi001 ceph-mon[29099]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-06-18T02:34:57.377 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:56 smithi001 ceph-mon[29099]: osdmap e32: 5 total, 4 up, 5 in 2024-06-18T02:34:57.377 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:56 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-18T02:34:57.377 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:56 smithi001 ceph-mon[29099]: from='osd.4 [v2:172.21.15.28:6808/276884153,v1:172.21.15.28:6809/276884153]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-06-18T02:34:57.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:56 smithi001 ceph-mon[29099]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-06-18T02:34:57.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:56 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:34:57.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:56 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:34:58.198 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:57 smithi028 ceph-mon[35823]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-18T02:34:58.198 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:57 smithi028 ceph-mon[35823]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-06-18T02:34:58.198 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:57 smithi028 ceph-mon[35823]: osdmap e33: 5 total, 4 up, 5 in 2024-06-18T02:34:58.198 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:57 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-18T02:34:58.198 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:57 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-18T02:34:58.377 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:57 smithi001 ceph-mon[29099]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-18T02:34:58.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:57 smithi001 ceph-mon[29099]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-06-18T02:34:58.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:57 smithi001 ceph-mon[29099]: osdmap e33: 5 total, 4 up, 5 in 2024-06-18T02:34:58.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:57 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-18T02:34:58.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:57 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-18T02:34:58.714 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph orch daemon add osd smithi028:vg_nvme/lv_2 2024-06-18T02:34:58.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:58 smithi028 ceph-mon[35823]: purged_snaps scrub starts 2024-06-18T02:34:58.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:58 smithi028 ceph-mon[35823]: purged_snaps scrub ok 2024-06-18T02:34:58.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:58 smithi028 ceph-mon[35823]: osd.4 [v2:172.21.15.28:6808/276884153,v1:172.21.15.28:6809/276884153] boot 2024-06-18T02:34:58.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:58 smithi028 ceph-mon[35823]: osdmap e34: 5 total, 5 up, 5 in 2024-06-18T02:34:58.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:58 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-18T02:34:59.000 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi028/config 2024-06-18T02:34:59.377 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:58 smithi001 ceph-mon[29099]: purged_snaps scrub starts 2024-06-18T02:34:59.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:58 smithi001 ceph-mon[29099]: purged_snaps scrub ok 2024-06-18T02:34:59.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:58 smithi001 ceph-mon[29099]: osd.4 [v2:172.21.15.28:6808/276884153,v1:172.21.15.28:6809/276884153] boot 2024-06-18T02:34:59.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:58 smithi001 ceph-mon[29099]: osdmap e34: 5 total, 5 up, 5 in 2024-06-18T02:34:59.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:58 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-18T02:35:00.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:59 smithi028 ceph-mon[35823]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-18T02:35:00.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:34:59 smithi028 ceph-mon[35823]: osdmap e35: 5 total, 5 up, 5 in 2024-06-18T02:35:00.377 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:59 smithi001 ceph-mon[29099]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-18T02:35:00.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:34:59 smithi001 ceph-mon[29099]: osdmap e35: 5 total, 5 up, 5 in 2024-06-18T02:35:01.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:01.099+0000 7f34e429f640 1 -- 172.21.15.28:0/721963728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34dc0fd550 msgr2=0x7f34dc0fd9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:35:01.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:01.099+0000 7f34e429f640 1 --2- 172.21.15.28:0/721963728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34dc0fd550 0x7f34dc0fd9c0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f34cc0048d0 tx=0x7f34cc009da0 comp rx=0 tx=0).stop 2024-06-18T02:35:01.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:01.099+0000 7f34e429f640 1 -- 172.21.15.28:0/721963728 shutdown_connections 2024-06-18T02:35:01.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:01.099+0000 7f34e429f640 1 --2- 172.21.15.28:0/721963728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34dc0fd550 0x7f34dc0fd9c0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:01.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:01.099+0000 7f34e429f640 1 --2- 172.21.15.28:0/721963728 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34dc0fc280 0x7f34dc0fc690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:01.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:01.099+0000 7f34e429f640 1 -- 172.21.15.28:0/721963728 >> 172.21.15.28:0/721963728 conn(0x7f34dc0f7a40 msgr2=0x7f34dc0f9e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:35:01.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:01.099+0000 7f34e429f640 1 -- 172.21.15.28:0/721963728 shutdown_connections 2024-06-18T02:35:01.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:01.099+0000 7f34e429f640 1 -- 172.21.15.28:0/721963728 wait complete. 2024-06-18T02:35:01.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:01.099+0000 7f34e429f640 1 Processor -- start 2024-06-18T02:35:01.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:01.100+0000 7f34e429f640 1 -- start start 2024-06-18T02:35:01.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:01.100+0000 7f34e429f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34dc0fc280 0x7f34dc0712d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:35:01.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:01.100+0000 7f34e429f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34dc0fd550 0x7f34dc06f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:35:01.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:01.100+0000 7f34e429f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34dc06fde0 con 0x7f34dc0fc280 2024-06-18T02:35:01.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:01.100+0000 7f34e429f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34dc06ff20 con 0x7f34dc0fd550 2024-06-18T02:35:01.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:01.100+0000 7f34e183a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34dc0fd550 0x7f34dc06f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:35:01.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:01.100+0000 7f34e183a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34dc0fd550 0x7f34dc06f7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56044/0 (socket says 172.21.15.28:56044) 2024-06-18T02:35:01.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:01.100+0000 7f34e183a640 1 -- 172.21.15.28:0/1187476878 learned_addr learned my addr 172.21.15.28:0/1187476878 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-06-18T02:35:01.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:01.100+0000 7f34e203b640 1 --2- 172.21.15.28:0/1187476878 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34dc0fc280 0x7f34dc0712d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:35:01.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:01.100+0000 7f34e183a640 1 -- 172.21.15.28:0/1187476878 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34dc0fc280 msgr2=0x7f34dc0712d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:35:01.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:01.100+0000 7f34e183a640 1 --2- 172.21.15.28:0/1187476878 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34dc0fc280 0x7f34dc0712d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:01.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:01.100+0000 7f34e183a640 1 -- 172.21.15.28:0/1187476878 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34cc005040 con 0x7f34dc0fd550 2024-06-18T02:35:01.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:01.100+0000 7f34e183a640 1 --2- 172.21.15.28:0/1187476878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34dc0fd550 0x7f34dc06f7c0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f34cc003080 tx=0x7f34cc009da0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:35:01.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:01.101+0000 7f34d2ffd640 1 -- 172.21.15.28:0/1187476878 <== mon.1 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f34cc038070 con 0x7f34dc0fd550 2024-06-18T02:35:01.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:01.101+0000 7f34d2ffd640 1 -- 172.21.15.28:0/1187476878 <== mon.1 v2:172.21.15.28:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f34cc032970 con 0x7f34dc0fd550 2024-06-18T02:35:01.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:01.101+0000 7f34e429f640 1 -- 172.21.15.28:0/1187476878 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34dc0701a0 con 0x7f34dc0fd550 2024-06-18T02:35:01.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:01.101+0000 7f34d2ffd640 1 -- 172.21.15.28:0/1187476878 <== mon.1 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f34cc033900 con 0x7f34dc0fd550 2024-06-18T02:35:01.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:01.101+0000 7f34e429f640 1 -- 172.21.15.28:0/1187476878 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34dc114f50 con 0x7f34dc0fd550 2024-06-18T02:35:01.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:01.102+0000 7f34e429f640 1 -- 172.21.15.28:0/1187476878 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34dc0fb6e0 con 0x7f34dc0fd550 2024-06-18T02:35:01.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:01.102+0000 7f34d2ffd640 1 -- 172.21.15.28:0/1187476878 <== mon.1 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7f34cc030020 con 0x7f34dc0fd550 2024-06-18T02:35:01.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:01.102+0000 7f34d2ffd640 1 --2- 172.21.15.28:0/1187476878 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f34c8065430 0x7f34c80678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:35:01.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:01.102+0000 7f34d2ffd640 1 -- 172.21.15.28:0/1187476878 <== mon.1 v2:172.21.15.28:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4202+0+0 (secure 0 0 0) 0x7f34cc0a2290 con 0x7f34dc0fd550 2024-06-18T02:35:01.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:01.103+0000 7f34e203b640 1 --2- 172.21.15.28:0/1187476878 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f34c8065430 0x7f34c80678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:35:01.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:01.103+0000 7f34e203b640 1 --2- 172.21.15.28:0/1187476878 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f34c8065430 0x7f34c80678e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f34dc071080 tx=0x7f34d8009120 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:35:01.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:01.107+0000 7f34d2ffd640 1 -- 172.21.15.28:0/1187476878 <== mon.1 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f34cc070a20 con 0x7f34dc0fd550 2024-06-18T02:35:01.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:01.290+0000 7f34e429f640 1 -- 172.21.15.28:0/1187476878 --> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f34dc101870 con 0x7f34c8065430 2024-06-18T02:35:01.875 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:01 smithi028 ceph-mon[35823]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-18T02:35:01.876 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:01 smithi028 ceph-mon[35823]: Detected new or changed devices on smithi028 2024-06-18T02:35:01.876 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:01 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:35:01.876 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:01 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:35:01.876 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:01 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-18T02:35:01.876 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:01 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-18T02:35:01.876 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:01 smithi028 ceph-mon[35823]: Adjusting osd_memory_target on smithi028 to 7970M 2024-06-18T02:35:01.876 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:01 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:35:01.876 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:01 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:35:01.876 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:01 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T02:35:01.876 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:01 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:35:01.876 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:01 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:35:01.876 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:01 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:35:01.877 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:01 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T02:35:01.877 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:01 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:35:01.877 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:01 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-18T02:35:01.877 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:01 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-18T02:35:01.877 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:01 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:35:01.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:01 smithi001 ceph-mon[29099]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-18T02:35:01.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:01 smithi001 ceph-mon[29099]: Detected new or changed devices on smithi028 2024-06-18T02:35:01.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:01 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:35:01.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:01 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:35:01.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:01 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-18T02:35:01.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:01 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-18T02:35:01.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:01 smithi001 ceph-mon[29099]: Adjusting osd_memory_target on smithi028 to 7970M 2024-06-18T02:35:01.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:01 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:35:01.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:01 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:35:01.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:01 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T02:35:01.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:01 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:35:01.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:01 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:35:01.879 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:01 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:35:01.879 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:01 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T02:35:01.879 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:01 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:35:01.879 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:01 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-18T02:35:01.879 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:01 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-18T02:35:01.879 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:01 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:35:02.737 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:02 smithi028 ceph-mon[35823]: from='client.24177 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-18T02:35:02.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:02 smithi001 ceph-mon[29099]: from='client.24177 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-18T02:35:03.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:03 smithi001 ceph-mon[29099]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-18T02:35:03.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:03 smithi001 ceph-mon[29099]: from='client.? 172.21.15.28:0/1495835457' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a2bf6f7d-170e-4b63-bf7a-8562ad2b66a8"}]: dispatch 2024-06-18T02:35:03.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:03 smithi001 ceph-mon[29099]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a2bf6f7d-170e-4b63-bf7a-8562ad2b66a8"}]: dispatch 2024-06-18T02:35:03.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:03 smithi001 ceph-mon[29099]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a2bf6f7d-170e-4b63-bf7a-8562ad2b66a8"}]': finished 2024-06-18T02:35:03.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:03 smithi001 ceph-mon[29099]: osdmap e36: 6 total, 5 up, 6 in 2024-06-18T02:35:03.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:03 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-18T02:35:03.989 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:03 smithi028 ceph-mon[35823]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-18T02:35:03.990 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:03 smithi028 ceph-mon[35823]: from='client.? 172.21.15.28:0/1495835457' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a2bf6f7d-170e-4b63-bf7a-8562ad2b66a8"}]: dispatch 2024-06-18T02:35:03.990 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:03 smithi028 ceph-mon[35823]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a2bf6f7d-170e-4b63-bf7a-8562ad2b66a8"}]: dispatch 2024-06-18T02:35:03.990 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:03 smithi028 ceph-mon[35823]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a2bf6f7d-170e-4b63-bf7a-8562ad2b66a8"}]': finished 2024-06-18T02:35:03.990 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:03 smithi028 ceph-mon[35823]: osdmap e36: 6 total, 5 up, 6 in 2024-06-18T02:35:03.990 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:03 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-18T02:35:04.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:04 smithi001 ceph-mon[29099]: from='client.? 172.21.15.28:0/3790309364' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-18T02:35:04.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:04 smithi028 ceph-mon[35823]: from='client.? 172.21.15.28:0/3790309364' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-18T02:35:05.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:05 smithi001 ceph-mon[29099]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-18T02:35:05.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:05 smithi028 ceph-mon[35823]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-18T02:35:06.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:06 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:35:06.973 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:06 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:35:07.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:07 smithi001 ceph-mon[29099]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-18T02:35:07.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:07 smithi028 ceph-mon[35823]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-18T02:35:09.876 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:09 smithi028 ceph-mon[35823]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-18T02:35:09.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:09 smithi001 ceph-mon[29099]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-18T02:35:10.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:10 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-06-18T02:35:10.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:10 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:35:10.897 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:10 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-06-18T02:35:10.897 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:10 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:35:11.900 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:11 smithi028 ceph-mon[35823]: Deploying daemon osd.5 on smithi028 2024-06-18T02:35:11.900 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:11 smithi028 ceph-mon[35823]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-18T02:35:12.127 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:11 smithi001 ceph-mon[29099]: Deploying daemon osd.5 on smithi028 2024-06-18T02:35:12.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:11 smithi001 ceph-mon[29099]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-18T02:35:13.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:13 smithi001 ceph-mon[29099]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-18T02:35:13.896 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:13 smithi028 ceph-mon[35823]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-18T02:35:15.754 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:15 smithi028 ceph-mon[35823]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-18T02:35:15.754 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:15 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:35:15.754 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:15 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:35:15.754 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:15 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:35:16.127 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:15 smithi001 ceph-mon[29099]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-18T02:35:16.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:15 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:35:16.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:15 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:35:16.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:15 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:35:17.921 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:17 smithi028 ceph-mon[35823]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-18T02:35:18.127 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:17 smithi001 ceph-mon[29099]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-18T02:35:18.958 INFO:teuthology.orchestra.run.smithi028.stdout:Created osd(s) 5 on host 'smithi028' 2024-06-18T02:35:18.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:18.958+0000 7f34d2ffd640 1 -- 172.21.15.28:0/1187476878 <== mgr.14216 v2:172.21.15.1:6800/729353917 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f34dc101870 con 0x7f34c8065430 2024-06-18T02:35:18.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:18.960+0000 7f34e429f640 1 -- 172.21.15.28:0/1187476878 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f34c8065430 msgr2=0x7f34c80678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:35:18.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:18.960+0000 7f34e429f640 1 --2- 172.21.15.28:0/1187476878 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f34c8065430 0x7f34c80678e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f34dc071080 tx=0x7f34d8009120 comp rx=0 tx=0).stop 2024-06-18T02:35:18.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:18.960+0000 7f34e429f640 1 -- 172.21.15.28:0/1187476878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34dc0fd550 msgr2=0x7f34dc06f7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:35:18.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:18.960+0000 7f34e429f640 1 --2- 172.21.15.28:0/1187476878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34dc0fd550 0x7f34dc06f7c0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f34cc003080 tx=0x7f34cc009da0 comp rx=0 tx=0).stop 2024-06-18T02:35:18.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:18.960+0000 7f34e429f640 1 -- 172.21.15.28:0/1187476878 shutdown_connections 2024-06-18T02:35:18.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:18.960+0000 7f34e429f640 1 --2- 172.21.15.28:0/1187476878 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f34c8065430 0x7f34c80678e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:18.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:18.960+0000 7f34e429f640 1 --2- 172.21.15.28:0/1187476878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34dc0fd550 0x7f34dc06f7c0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:18.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:18.960+0000 7f34e429f640 1 --2- 172.21.15.28:0/1187476878 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34dc0fc280 0x7f34dc0712d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:18.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:18.960+0000 7f34e429f640 1 -- 172.21.15.28:0/1187476878 >> 172.21.15.28:0/1187476878 conn(0x7f34dc0f7a40 msgr2=0x7f34dc0f9110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:35:18.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:18.960+0000 7f34e429f640 1 -- 172.21.15.28:0/1187476878 shutdown_connections 2024-06-18T02:35:18.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:18.960+0000 7f34e429f640 1 -- 172.21.15.28:0/1187476878 wait complete. 2024-06-18T02:35:19.568 DEBUG:teuthology.orchestra.run.smithi028:osd.5> sudo journalctl -f -n 0 -u ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968@osd.5.service 2024-06-18T02:35:19.570 INFO:tasks.cephadm:Waiting for 6 OSDs to come up... 2024-06-18T02:35:19.570 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph osd stat -f json 2024-06-18T02:35:19.863 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:35:19.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:19 smithi001 ceph-mon[29099]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-18T02:35:19.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:19 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:35:19.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:19 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:35:19.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:19 smithi001 ceph-mon[29099]: from='osd.5 [v2:172.21.15.28:6816/2643308264,v1:172.21.15.28:6817/2643308264]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-18T02:35:19.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:19 smithi001 ceph-mon[29099]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-18T02:35:19.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:19 smithi028 ceph-mon[35823]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-18T02:35:19.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:19 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:35:19.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:19 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:35:19.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:19 smithi028 ceph-mon[35823]: from='osd.5 [v2:172.21.15.28:6816/2643308264,v1:172.21.15.28:6817/2643308264]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-18T02:35:19.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:19 smithi028 ceph-mon[35823]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-18T02:35:20.819 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:20 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:35:20.819 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:20 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:35:20.819 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:20 smithi028 ceph-mon[35823]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-06-18T02:35:20.819 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:20 smithi028 ceph-mon[35823]: osdmap e37: 6 total, 5 up, 6 in 2024-06-18T02:35:20.819 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:20 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-18T02:35:20.819 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:20 smithi028 ceph-mon[35823]: from='osd.5 [v2:172.21.15.28:6816/2643308264,v1:172.21.15.28:6817/2643308264]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-06-18T02:35:20.819 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:20 smithi028 ceph-mon[35823]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-06-18T02:35:21.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:20 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:35:21.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:20 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:35:21.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:20 smithi001 ceph-mon[29099]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-06-18T02:35:21.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:20 smithi001 ceph-mon[29099]: osdmap e37: 6 total, 5 up, 6 in 2024-06-18T02:35:21.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:20 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-18T02:35:21.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:20 smithi001 ceph-mon[29099]: from='osd.5 [v2:172.21.15.28:6816/2643308264,v1:172.21.15.28:6817/2643308264]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-06-18T02:35:21.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:20 smithi001 ceph-mon[29099]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-06-18T02:35:21.155 INFO:journalctl@ceph.osd.5.smithi028.stdout:Jun 18 02:35:20 smithi028 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-5[54541]: 2024-06-18T02:35:20.962+0000 7fe4397e5640 -1 osd.5 0 waiting for initial osdmap 2024-06-18T02:35:21.155 INFO:journalctl@ceph.osd.5.smithi028.stdout:Jun 18 02:35:20 smithi028 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-5[54541]: 2024-06-18T02:35:20.970+0000 7fe434e1d640 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-18T02:35:21.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:21.633+0000 7f2bd2691640 1 -- 172.21.15.1:0/1128220598 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2bcc0fd900 msgr2=0x7f2bcc0fdd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:35:21.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:21.633+0000 7f2bd2691640 1 --2- 172.21.15.1:0/1128220598 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2bcc0fd900 0x7f2bcc0fdd10 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc0048d0 tx=0x7f2bbc009da0 comp rx=0 tx=0).stop 2024-06-18T02:35:21.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:21.633+0000 7f2bd2691640 1 -- 172.21.15.1:0/1128220598 shutdown_connections 2024-06-18T02:35:21.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:21.633+0000 7f2bd2691640 1 --2- 172.21.15.1:0/1128220598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bcc0fe250 0x7f2bcc0fa3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:21.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:21.633+0000 7f2bd2691640 1 --2- 172.21.15.1:0/1128220598 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2bcc0fd900 0x7f2bcc0fdd10 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:21.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:21.633+0000 7f2bd2691640 1 -- 172.21.15.1:0/1128220598 >> 172.21.15.1:0/1128220598 conn(0x7f2bcc0f6100 msgr2=0x7f2bcc0f8530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:35:21.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:21.633+0000 7f2bd2691640 1 -- 172.21.15.1:0/1128220598 shutdown_connections 2024-06-18T02:35:21.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:21.633+0000 7f2bd2691640 1 -- 172.21.15.1:0/1128220598 wait complete. 2024-06-18T02:35:21.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:21.634+0000 7f2bd2691640 1 Processor -- start 2024-06-18T02:35:21.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:21.634+0000 7f2bd2691640 1 -- start start 2024-06-18T02:35:21.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:21.634+0000 7f2bd2691640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2bcc0fd900 0x7f2bcc10cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:35:21.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:21.634+0000 7f2bd2691640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bcc0fe250 0x7f2bcc10d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:35:21.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:21.634+0000 7f2bd2691640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bcc10da50 con 0x7f2bcc0fd900 2024-06-18T02:35:21.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:21.634+0000 7f2bd2691640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bcc10e270 con 0x7f2bcc0fe250 2024-06-18T02:35:21.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:21.634+0000 7f2bcbfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2bcc0fd900 0x7f2bcc10cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:35:21.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:21.634+0000 7f2bcbfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2bcc0fd900 0x7f2bcc10cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33774/0 (socket says 172.21.15.1:33774) 2024-06-18T02:35:21.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:21.634+0000 7f2bcbfff640 1 -- 172.21.15.1:0/4086253018 learned_addr learned my addr 172.21.15.1:0/4086253018 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:35:21.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:21.635+0000 7f2bcb7fe640 1 --2- 172.21.15.1:0/4086253018 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bcc0fe250 0x7f2bcc10d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:35:21.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:21.635+0000 7f2bcbfff640 1 -- 172.21.15.1:0/4086253018 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bcc0fe250 msgr2=0x7f2bcc10d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:35:21.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:21.635+0000 7f2bcbfff640 1 --2- 172.21.15.1:0/4086253018 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bcc0fe250 0x7f2bcc10d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:21.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:21.635+0000 7f2bcbfff640 1 -- 172.21.15.1:0/4086253018 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bbc005040 con 0x7f2bcc0fd900 2024-06-18T02:35:21.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:21.635+0000 7f2bcbfff640 1 --2- 172.21.15.1:0/4086253018 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2bcc0fd900 0x7f2bcc10cec0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc003080 tx=0x7f2bbc009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:35:21.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:21.635+0000 7f2bc8ff9640 1 -- 172.21.15.1:0/4086253018 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2bbc038070 con 0x7f2bcc0fd900 2024-06-18T02:35:21.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:21.635+0000 7f2bc8ff9640 1 -- 172.21.15.1:0/4086253018 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2bbc032970 con 0x7f2bcc0fd900 2024-06-18T02:35:21.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:21.635+0000 7f2bc8ff9640 1 -- 172.21.15.1:0/4086253018 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2bbc033900 con 0x7f2bcc0fd900 2024-06-18T02:35:21.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:21.635+0000 7f2bd2691640 1 -- 172.21.15.1:0/4086253018 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bcc10e4a0 con 0x7f2bcc0fd900 2024-06-18T02:35:21.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:21.635+0000 7f2bd2691640 1 -- 172.21.15.1:0/4086253018 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bcc10e9e0 con 0x7f2bcc0fd900 2024-06-18T02:35:21.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:21.636+0000 7f2bd2691640 1 -- 172.21.15.1:0/4086253018 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bcc0f9da0 con 0x7f2bcc0fd900 2024-06-18T02:35:21.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:21.637+0000 7f2bc8ff9640 1 -- 172.21.15.1:0/4086253018 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7f2bbc030020 con 0x7f2bcc0fd900 2024-06-18T02:35:21.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:21.637+0000 7f2bc8ff9640 1 --2- 172.21.15.1:0/4086253018 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f2bb4065430 0x7f2bb40678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:35:21.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:21.637+0000 7f2bc8ff9640 1 -- 172.21.15.1:0/4086253018 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4354+0+0 (secure 0 0 0) 0x7f2bbc0a23e0 con 0x7f2bcc0fd900 2024-06-18T02:35:21.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:21.637+0000 7f2bcb7fe640 1 --2- 172.21.15.1:0/4086253018 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f2bb4065430 0x7f2bb40678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:35:21.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:21.638+0000 7f2bcb7fe640 1 --2- 172.21.15.1:0/4086253018 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f2bb4065430 0x7f2bb40678e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc10cc70 tx=0x7f2bc0009040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:35:21.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:21.642+0000 7f2bc8ff9640 1 -- 172.21.15.1:0/4086253018 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f2bbc070a20 con 0x7f2bcc0fd900 2024-06-18T02:35:21.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:21.820+0000 7f2bd2691640 1 -- 172.21.15.1:0/4086253018 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f2bcc04b480 con 0x7f2bcc0fd900 2024-06-18T02:35:21.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:21.820+0000 7f2bc8ff9640 1 -- 172.21.15.1:0/4086253018 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v38) v1 ==== 74+0+130 (secure 0 0 0) 0x7f2bbc074040 con 0x7f2bcc0fd900 2024-06-18T02:35:21.820 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:35:21.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:21.822+0000 7f2bd2691640 1 -- 172.21.15.1:0/4086253018 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f2bb4065430 msgr2=0x7f2bb40678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:35:21.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:21.822+0000 7f2bd2691640 1 --2- 172.21.15.1:0/4086253018 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f2bb4065430 0x7f2bb40678e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc10cc70 tx=0x7f2bc0009040 comp rx=0 tx=0).stop 2024-06-18T02:35:21.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:21.822+0000 7f2bd2691640 1 -- 172.21.15.1:0/4086253018 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2bcc0fd900 msgr2=0x7f2bcc10cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:35:21.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:21.822+0000 7f2bd2691640 1 --2- 172.21.15.1:0/4086253018 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2bcc0fd900 0x7f2bcc10cec0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc003080 tx=0x7f2bbc009da0 comp rx=0 tx=0).stop 2024-06-18T02:35:21.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:21.822+0000 7f2bd2691640 1 -- 172.21.15.1:0/4086253018 shutdown_connections 2024-06-18T02:35:21.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:21.822+0000 7f2bd2691640 1 --2- 172.21.15.1:0/4086253018 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f2bb4065430 0x7f2bb40678e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:21.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:21.822+0000 7f2bd2691640 1 --2- 172.21.15.1:0/4086253018 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bcc0fe250 0x7f2bcc10d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:21.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:21.822+0000 7f2bd2691640 1 --2- 172.21.15.1:0/4086253018 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2bcc0fd900 0x7f2bcc10cec0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:21.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:21.822+0000 7f2bd2691640 1 -- 172.21.15.1:0/4086253018 >> 172.21.15.1:0/4086253018 conn(0x7f2bcc0f6100 msgr2=0x7f2bcc0f77a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:35:21.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:21.822+0000 7f2bd2691640 1 -- 172.21.15.1:0/4086253018 shutdown_connections 2024-06-18T02:35:21.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:21.822+0000 7f2bd2691640 1 -- 172.21.15.1:0/4086253018 wait complete. 2024-06-18T02:35:22.127 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:21 smithi001 ceph-mon[29099]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-18T02:35:22.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:21 smithi001 ceph-mon[29099]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-06-18T02:35:22.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:21 smithi001 ceph-mon[29099]: osdmap e38: 6 total, 5 up, 6 in 2024-06-18T02:35:22.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:21 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-18T02:35:22.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:21 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-18T02:35:22.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:21 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:35:22.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:21 smithi028 ceph-mon[35823]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-18T02:35:22.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:21 smithi028 ceph-mon[35823]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-06-18T02:35:22.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:21 smithi028 ceph-mon[35823]: osdmap e38: 6 total, 5 up, 6 in 2024-06-18T02:35:22.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:21 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-18T02:35:22.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:21 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-18T02:35:22.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:21 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:35:22.590 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":38,"num_osds":6,"num_up_osds":5,"osd_up_since":1718678097,"num_in_osds":6,"osd_in_since":1718678103,"num_remapped_pgs":0} 2024-06-18T02:35:22.866 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:22 smithi001 ceph-mon[29099]: purged_snaps scrub starts 2024-06-18T02:35:22.866 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:22 smithi001 ceph-mon[29099]: purged_snaps scrub ok 2024-06-18T02:35:22.867 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:22 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/4086253018' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-06-18T02:35:22.867 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:22 smithi001 ceph-mon[29099]: osd.5 [v2:172.21.15.28:6816/2643308264,v1:172.21.15.28:6817/2643308264] boot 2024-06-18T02:35:22.867 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:22 smithi001 ceph-mon[29099]: osdmap e39: 6 total, 6 up, 6 in 2024-06-18T02:35:22.867 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:22 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-18T02:35:22.922 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:22 smithi028 ceph-mon[35823]: purged_snaps scrub starts 2024-06-18T02:35:22.922 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:22 smithi028 ceph-mon[35823]: purged_snaps scrub ok 2024-06-18T02:35:22.923 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:22 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/4086253018' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-06-18T02:35:22.923 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:22 smithi028 ceph-mon[35823]: osd.5 [v2:172.21.15.28:6816/2643308264,v1:172.21.15.28:6817/2643308264] boot 2024-06-18T02:35:22.923 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:22 smithi028 ceph-mon[35823]: osdmap e39: 6 total, 6 up, 6 in 2024-06-18T02:35:22.923 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:22 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-18T02:35:23.591 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph osd stat -f json 2024-06-18T02:35:23.861 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:35:23.871 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:23 smithi001 ceph-mon[29099]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-18T02:35:23.871 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:23 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:35:23.871 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:23 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:35:23.871 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:23 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-18T02:35:23.871 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:23 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-18T02:35:23.871 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:23 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-18T02:35:23.871 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:23 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:35:23.871 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:23 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:35:23.871 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:23 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T02:35:23.871 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:23 smithi001 ceph-mon[29099]: osdmap e40: 6 total, 6 up, 6 in 2024-06-18T02:35:23.872 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:23 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:35:23.872 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:23 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:35:23.872 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:23 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:35:23.872 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:23 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T02:35:23.872 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:23 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:35:24.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:23 smithi028 ceph-mon[35823]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-18T02:35:24.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:23 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:35:24.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:23 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:35:24.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:23 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-18T02:35:24.239 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:23 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-18T02:35:24.239 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:23 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-18T02:35:24.239 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:23 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:35:24.239 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:23 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:35:24.239 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:23 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T02:35:24.239 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:23 smithi028 ceph-mon[35823]: osdmap e40: 6 total, 6 up, 6 in 2024-06-18T02:35:24.240 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:23 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:35:24.240 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:23 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:35:24.240 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:23 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:35:24.240 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:23 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T02:35:24.240 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:23 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:35:25.019 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:24 smithi001 ceph-mon[29099]: Detected new or changed devices on smithi028 2024-06-18T02:35:25.020 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:24 smithi001 ceph-mon[29099]: Adjusting osd_memory_target on smithi028 to 5313M 2024-06-18T02:35:25.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:24 smithi028 ceph-mon[35823]: Detected new or changed devices on smithi028 2024-06-18T02:35:25.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:24 smithi028 ceph-mon[35823]: Adjusting osd_memory_target on smithi028 to 5313M 2024-06-18T02:35:25.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:25.786+0000 7f5517613640 1 -- 172.21.15.1:0/1588881240 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55100fe250 msgr2=0x7f55100fa3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:35:25.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:25.786+0000 7f5517613640 1 --2- 172.21.15.1:0/1588881240 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55100fe250 0x7f55100fa3b0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f55000048d0 tx=0x7f5500009da0 comp rx=0 tx=0).stop 2024-06-18T02:35:25.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:25.786+0000 7f5517613640 1 -- 172.21.15.1:0/1588881240 shutdown_connections 2024-06-18T02:35:25.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:25.786+0000 7f5517613640 1 --2- 172.21.15.1:0/1588881240 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55100fe250 0x7f55100fa3b0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:25.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:25.786+0000 7f5517613640 1 --2- 172.21.15.1:0/1588881240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55100fd900 0x7f55100fdd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:25.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:25.786+0000 7f5517613640 1 -- 172.21.15.1:0/1588881240 >> 172.21.15.1:0/1588881240 conn(0x7f55100f6100 msgr2=0x7f55100f8530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:35:25.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:25.786+0000 7f5517613640 1 -- 172.21.15.1:0/1588881240 shutdown_connections 2024-06-18T02:35:25.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:25.786+0000 7f5517613640 1 -- 172.21.15.1:0/1588881240 wait complete. 2024-06-18T02:35:25.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:25.787+0000 7f5517613640 1 Processor -- start 2024-06-18T02:35:25.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:25.787+0000 7f5517613640 1 -- start start 2024-06-18T02:35:25.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:25.787+0000 7f5517613640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55100fd900 0x7f551010cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:35:25.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:25.787+0000 7f5517613640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55100fe250 0x7f551010d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:35:25.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:25.787+0000 7f5517613640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f551010da50 con 0x7f55100fe250 2024-06-18T02:35:25.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:25.787+0000 7f5517613640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f551010e270 con 0x7f55100fd900 2024-06-18T02:35:25.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:25.787+0000 7f5514bae640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55100fe250 0x7f551010d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:35:25.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:25.787+0000 7f5514bae640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55100fe250 0x7f551010d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43388/0 (socket says 172.21.15.1:43388) 2024-06-18T02:35:25.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:25.787+0000 7f5514bae640 1 -- 172.21.15.1:0/3653662983 learned_addr learned my addr 172.21.15.1:0/3653662983 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:35:25.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:25.787+0000 7f55153af640 1 --2- 172.21.15.1:0/3653662983 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55100fd900 0x7f551010cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:35:25.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:25.787+0000 7f5514bae640 1 -- 172.21.15.1:0/3653662983 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55100fd900 msgr2=0x7f551010cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:35:25.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:25.787+0000 7f5514bae640 1 --2- 172.21.15.1:0/3653662983 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55100fd900 0x7f551010cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:25.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:25.787+0000 7f5514bae640 1 -- 172.21.15.1:0/3653662983 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5500005040 con 0x7f55100fe250 2024-06-18T02:35:25.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:25.787+0000 7f55153af640 1 --2- 172.21.15.1:0/3653662983 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55100fd900 0x7f551010cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-18T02:35:25.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:25.788+0000 7f5514bae640 1 --2- 172.21.15.1:0/3653662983 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55100fe250 0x7f551010d400 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5500000c00 tx=0x7f5500009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:35:25.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:25.788+0000 7f5505ffb640 1 -- 172.21.15.1:0/3653662983 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5500038070 con 0x7f55100fe250 2024-06-18T02:35:25.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:25.788+0000 7f5505ffb640 1 -- 172.21.15.1:0/3653662983 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5500032970 con 0x7f55100fe250 2024-06-18T02:35:25.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:25.788+0000 7f5517613640 1 -- 172.21.15.1:0/3653662983 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f551010e4a0 con 0x7f55100fe250 2024-06-18T02:35:25.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:25.788+0000 7f5505ffb640 1 -- 172.21.15.1:0/3653662983 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f55000338e0 con 0x7f55100fe250 2024-06-18T02:35:25.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:25.788+0000 7f5517613640 1 -- 172.21.15.1:0/3653662983 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f551010e9e0 con 0x7f55100fe250 2024-06-18T02:35:25.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:25.789+0000 7f5517613640 1 -- 172.21.15.1:0/3653662983 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55100f9da0 con 0x7f55100fe250 2024-06-18T02:35:25.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:25.790+0000 7f5505ffb640 1 -- 172.21.15.1:0/3653662983 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7f5500030020 con 0x7f55100fe250 2024-06-18T02:35:25.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:25.790+0000 7f5505ffb640 1 --2- 172.21.15.1:0/3653662983 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f54fc065430 0x7f54fc0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:35:25.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:25.790+0000 7f5505ffb640 1 -- 172.21.15.1:0/3653662983 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f55000a2340 con 0x7f55100fe250 2024-06-18T02:35:25.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:25.790+0000 7f55153af640 1 --2- 172.21.15.1:0/3653662983 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f54fc065430 0x7f54fc0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:35:25.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:25.790+0000 7f55153af640 1 --2- 172.21.15.1:0/3653662983 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f54fc065430 0x7f54fc0678e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f550c007fd0 tx=0x7f550c009e90 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:35:25.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:25.794+0000 7f5505ffb640 1 -- 172.21.15.1:0/3653662983 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f55000707d0 con 0x7f55100fe250 2024-06-18T02:35:25.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:25 smithi001 ceph-mon[29099]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:35:25.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:25.968+0000 7f5517613640 1 -- 172.21.15.1:0/3653662983 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f551004b480 con 0x7f55100fe250 2024-06-18T02:35:25.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:25.969+0000 7f5505ffb640 1 -- 172.21.15.1:0/3653662983 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v40) v1 ==== 74+0+130 (secure 0 0 0) 0x7f5500041090 con 0x7f55100fe250 2024-06-18T02:35:25.968 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:35:25.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:25.971+0000 7f5517613640 1 -- 172.21.15.1:0/3653662983 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f54fc065430 msgr2=0x7f54fc0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:35:25.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:25.971+0000 7f5517613640 1 --2- 172.21.15.1:0/3653662983 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f54fc065430 0x7f54fc0678e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f550c007fd0 tx=0x7f550c009e90 comp rx=0 tx=0).stop 2024-06-18T02:35:25.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:25.971+0000 7f5517613640 1 -- 172.21.15.1:0/3653662983 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55100fe250 msgr2=0x7f551010d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:35:25.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:25.971+0000 7f5517613640 1 --2- 172.21.15.1:0/3653662983 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55100fe250 0x7f551010d400 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5500000c00 tx=0x7f5500009da0 comp rx=0 tx=0).stop 2024-06-18T02:35:25.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:25.971+0000 7f5517613640 1 -- 172.21.15.1:0/3653662983 shutdown_connections 2024-06-18T02:35:25.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:25.971+0000 7f5517613640 1 --2- 172.21.15.1:0/3653662983 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f54fc065430 0x7f54fc0678e0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:25.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:25.971+0000 7f5517613640 1 --2- 172.21.15.1:0/3653662983 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f55100fe250 0x7f551010d400 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:25.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:25.971+0000 7f5517613640 1 --2- 172.21.15.1:0/3653662983 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55100fd900 0x7f551010cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:25.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:25.971+0000 7f5517613640 1 -- 172.21.15.1:0/3653662983 >> 172.21.15.1:0/3653662983 conn(0x7f55100f6100 msgr2=0x7f55100f77c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:35:25.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:25.971+0000 7f5517613640 1 -- 172.21.15.1:0/3653662983 shutdown_connections 2024-06-18T02:35:25.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:25.971+0000 7f5517613640 1 -- 172.21.15.1:0/3653662983 wait complete. 2024-06-18T02:35:26.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:25 smithi028 ceph-mon[35823]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:35:26.734 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":40,"num_osds":6,"num_up_osds":6,"osd_up_since":1718678121,"num_in_osds":6,"osd_in_since":1718678103,"num_remapped_pgs":0} 2024-06-18T02:35:26.735 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph osd dump --format=json 2024-06-18T02:35:27.005 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:35:27.127 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:26 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/3653662983' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-06-18T02:35:27.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:26 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/3653662983' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-06-18T02:35:28.069 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:27 smithi001 ceph-mon[29099]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:35:28.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:27 smithi028 ceph-mon[35823]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:35:28.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:28.680+0000 7f2334a5e640 1 -- 172.21.15.1:0/1295556848 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2330101cb0 msgr2=0x7f23300fb230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:35:28.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:28.680+0000 7f2334a5e640 1 --2- 172.21.15.1:0/1295556848 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2330101cb0 0x7f23300fb230 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f23180048d0 tx=0x7f2318009da0 comp rx=0 tx=0).stop 2024-06-18T02:35:28.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:28.680+0000 7f2334a5e640 1 -- 172.21.15.1:0/1295556848 shutdown_connections 2024-06-18T02:35:28.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:28.680+0000 7f2334a5e640 1 --2- 172.21.15.1:0/1295556848 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2330101cb0 0x7f23300fb230 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:28.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:28.680+0000 7f2334a5e640 1 --2- 172.21.15.1:0/1295556848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2330101310 0x7f23301016e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:28.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:28.680+0000 7f2334a5e640 1 -- 172.21.15.1:0/1295556848 >> 172.21.15.1:0/1295556848 conn(0x7f23300f6d90 msgr2=0x7f23300f91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:35:28.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:28.680+0000 7f2334a5e640 1 -- 172.21.15.1:0/1295556848 shutdown_connections 2024-06-18T02:35:28.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:28.681+0000 7f2334a5e640 1 -- 172.21.15.1:0/1295556848 wait complete. 2024-06-18T02:35:28.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:28.681+0000 7f2334a5e640 1 Processor -- start 2024-06-18T02:35:28.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:28.681+0000 7f2334a5e640 1 -- start start 2024-06-18T02:35:28.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:28.681+0000 7f2334a5e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2330101310 0x7f2330113130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:35:28.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:28.681+0000 7f2334a5e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2330101cb0 0x7f2330113670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:35:28.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:28.681+0000 7f2334a5e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2330113c70 con 0x7f2330101cb0 2024-06-18T02:35:28.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:28.681+0000 7f2334a5e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f233010d0b0 con 0x7f2330101310 2024-06-18T02:35:28.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:28.681+0000 7f2327fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2330101cb0 0x7f2330113670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:35:28.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:28.681+0000 7f2327fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2330101cb0 0x7f2330113670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43404/0 (socket says 172.21.15.1:43404) 2024-06-18T02:35:28.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:28.681+0000 7f2327fff640 1 -- 172.21.15.1:0/474469037 learned_addr learned my addr 172.21.15.1:0/474469037 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:35:28.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:28.681+0000 7f232e59c640 1 --2- 172.21.15.1:0/474469037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2330101310 0x7f2330113130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:35:28.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:28.682+0000 7f2327fff640 1 -- 172.21.15.1:0/474469037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2330101310 msgr2=0x7f2330113130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:35:28.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:28.682+0000 7f2327fff640 1 --2- 172.21.15.1:0/474469037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2330101310 0x7f2330113130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:28.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:28.682+0000 7f2327fff640 1 -- 172.21.15.1:0/474469037 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2318005040 con 0x7f2330101cb0 2024-06-18T02:35:28.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:28.682+0000 7f2327fff640 1 --2- 172.21.15.1:0/474469037 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2330101cb0 0x7f2330113670 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f23180043d0 tx=0x7f2318009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:35:28.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:28.682+0000 7f2326ffd640 1 -- 172.21.15.1:0/474469037 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2318038070 con 0x7f2330101cb0 2024-06-18T02:35:28.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:28.682+0000 7f2326ffd640 1 -- 172.21.15.1:0/474469037 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2318032970 con 0x7f2330101cb0 2024-06-18T02:35:28.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:28.682+0000 7f2334a5e640 1 -- 172.21.15.1:0/474469037 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f233010d330 con 0x7f2330101cb0 2024-06-18T02:35:28.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:28.682+0000 7f2326ffd640 1 -- 172.21.15.1:0/474469037 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f231803da60 con 0x7f2330101cb0 2024-06-18T02:35:28.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:28.682+0000 7f2334a5e640 1 -- 172.21.15.1:0/474469037 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f233010d870 con 0x7f2330101cb0 2024-06-18T02:35:28.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:28.683+0000 7f2334a5e640 1 -- 172.21.15.1:0/474469037 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23300fa9f0 con 0x7f2330101cb0 2024-06-18T02:35:28.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:28.684+0000 7f2326ffd640 1 -- 172.21.15.1:0/474469037 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7f2318030020 con 0x7f2330101cb0 2024-06-18T02:35:28.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:28.684+0000 7f2326ffd640 1 --2- 172.21.15.1:0/474469037 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f231c065430 0x7f231c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:35:28.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:28.684+0000 7f2326ffd640 1 -- 172.21.15.1:0/474469037 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f23180a25b0 con 0x7f2330101cb0 2024-06-18T02:35:28.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:28.684+0000 7f232e59c640 1 --2- 172.21.15.1:0/474469037 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f231c065430 0x7f231c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:35:28.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:28.684+0000 7f232e59c640 1 --2- 172.21.15.1:0/474469037 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f231c065430 0x7f231c0678e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f2320004ed0 tx=0x7f232000b040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:35:28.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:28.688+0000 7f2326ffd640 1 -- 172.21.15.1:0/474469037 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f2318070a60 con 0x7f2330101cb0 2024-06-18T02:35:28.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:28.864+0000 7f2334a5e640 1 -- 172.21.15.1:0/474469037 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f233004b480 con 0x7f2330101cb0 2024-06-18T02:35:28.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:28.865+0000 7f2326ffd640 1 -- 172.21.15.1:0/474469037 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+10965 (secure 0 0 0) 0x7f2318074080 con 0x7f2330101cb0 2024-06-18T02:35:28.865 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:35:28.865 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":40,"fsid":"7eefb63e-2d1a-11ef-bc9d-c7b262605968","created":"2024-06-18T02:29:58.467010+0000","modified":"2024-06-18T02:35:22.963059+0000","last_up_change":"2024-06-18T02:35:21.961141+0000","last_in_change":"2024-06-18T02:35:03.501632+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-18T02:34:11.007633+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"2fc85253-858e-496d-a262-061526504dc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":797495043},{"type":"v1","addr":"172.21.15.1:6803","nonce":797495043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":797495043},{"type":"v1","addr":"172.21.15.1:6805","nonce":797495043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":797495043},{"type":"v1","addr":"172.21.15.1:6809","nonce":797495043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":797495043},{"type":"v1","addr":"172.21.15.1:6807","nonce":797495043}]},"public_addr":"172.21.15.1:6803/797495043","cluster_addr":"172.21.15.1:6805/797495043","heartbeat_back_addr":"172.21.15.1:6809/797495043","heartbeat_front_addr":"172.21.15.1:6807/797495043","state":["exists","up"]},{"osd":1,"uuid":"f9bb7834-4fa4-40b1-9b63-bd1a69a4e082","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":2182416687},{"type":"v1","addr":"172.21.15.1:6811","nonce":2182416687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":2182416687},{"type":"v1","addr":"172.21.15.1:6813","nonce":2182416687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":2182416687},{"type":"v1","addr":"172.21.15.1:6817","nonce":2182416687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":2182416687},{"type":"v1","addr":"172.21.15.1:6815","nonce":2182416687}]},"public_addr":"172.21.15.1:6811/2182416687","cluster_addr":"172.21.15.1:6813/2182416687","heartbeat_back_addr":"172.21.15.1:6817/2182416687","heartbeat_front_addr":"172.21.15.1:6815/2182416687","state":["exists","up"]},{"osd":2,"uuid":"89223bec-bba5-4744-960e-5396f04ddba1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":3682942580},{"type":"v1","addr":"172.21.15.1:6819","nonce":3682942580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3682942580},{"type":"v1","addr":"172.21.15.1:6821","nonce":3682942580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3682942580},{"type":"v1","addr":"172.21.15.1:6825","nonce":3682942580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":3682942580},{"type":"v1","addr":"172.21.15.1:6823","nonce":3682942580}]},"public_addr":"172.21.15.1:6819/3682942580","cluster_addr":"172.21.15.1:6821/3682942580","heartbeat_back_addr":"172.21.15.1:6825/3682942580","heartbeat_front_addr":"172.21.15.1:6823/3682942580","state":["exists","up"]},{"osd":3,"uuid":"33dd638d-10d2-432a-92ac-74b68b4a8311","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":664150046},{"type":"v1","addr":"172.21.15.28:6801","nonce":664150046}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":664150046},{"type":"v1","addr":"172.21.15.28:6803","nonce":664150046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":664150046},{"type":"v1","addr":"172.21.15.28:6807","nonce":664150046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":664150046},{"type":"v1","addr":"172.21.15.28:6805","nonce":664150046}]},"public_addr":"172.21.15.28:6801/664150046","cluster_addr":"172.21.15.28:6803/664150046","heartbeat_back_addr":"172.21.15.28:6807/664150046","heartbeat_front_addr":"172.21.15.28:6805/664150046","state":["exists","up"]},{"osd":4,"uuid":"2f4a6526-2f3e-417f-b02c-cf242e7eac07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":276884153},{"type":"v1","addr":"172.21.15.28:6809","nonce":276884153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":276884153},{"type":"v1","addr":"172.21.15.28:6811","nonce":276884153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":276884153},{"type":"v1","addr":"172.21.15.28:6815","nonce":276884153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":276884153},{"type":"v1","addr":"172.21.15.28:6813","nonce":276884153}]},"public_addr":"172.21.15.28:6809/276884153","cluster_addr":"172.21.15.28:6811/276884153","heartbeat_back_addr":"172.21.15.28:6815/276884153","heartbeat_front_addr":"172.21.15.28:6813/276884153","state":["exists","up"]},{"osd":5,"uuid":"a2bf6f7d-170e-4b63-bf7a-8562ad2b66a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2643308264},{"type":"v1","addr":"172.21.15.28:6817","nonce":2643308264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":2643308264},{"type":"v1","addr":"172.21.15.28:6819","nonce":2643308264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":2643308264},{"type":"v1","addr":"172.21.15.28:6823","nonce":2643308264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2643308264},{"type":"v1","addr":"172.21.15.28:6821","nonce":2643308264}]},"public_addr":"172.21.15.28:6817/2643308264","cluster_addr":"172.21.15.28:6819/2643308264","heartbeat_back_addr":"172.21.15.28:6823/2643308264","heartbeat_front_addr":"172.21.15.28:6821/2643308264","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-18T02:33:14.758941+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-18T02:33:41.399768+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-18T02:34:08.356814+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-18T02:34:31.488432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-18T02:34:56.105704+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-18T02:35:20.190437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/776743141":"2024-06-19T02:32:06.496885+0000","172.21.15.1:0/2875918960":"2024-06-19T02:32:06.496885+0000","172.21.15.1:6801/752464819":"2024-06-19T02:30:29.235259+0000","172.21.15.1:6801/4268462134":"2024-06-19T02:30:58.990073+0000","172.21.15.1:0/1466863637":"2024-06-19T02:30:29.235259+0000","172.21.15.1:0/1245204308":"2024-06-19T02:30:29.235259+0000","172.21.15.1:0/2844026642":"2024-06-19T02:30:58.990073+0000","172.21.15.1:0/138615605":"2024-06-19T02:30:58.990073+0000","172.21.15.1:0/2566240088":"2024-06-19T02:30:29.235259+0000","172.21.15.1:0/2788805455":"2024-06-19T02:30:58.990073+0000","172.21.15.1:6801/85550937":"2024-06-19T02:32:06.496885+0000","172.21.15.1:0/258317390":"2024-06-19T02:32:06.496885+0000","172.21.15.1:6800/752464819":"2024-06-19T02:30:29.235259+0000","172.21.15.1:6800/4268462134":"2024-06-19T02:30:58.990073+0000","172.21.15.1:6800/85550937":"2024-06-19T02:32:06.496885+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-18T02:35:28.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:28.867+0000 7f2334a5e640 1 -- 172.21.15.1:0/474469037 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f231c065430 msgr2=0x7f231c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:35:28.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:28.867+0000 7f2334a5e640 1 --2- 172.21.15.1:0/474469037 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f231c065430 0x7f231c0678e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f2320004ed0 tx=0x7f232000b040 comp rx=0 tx=0).stop 2024-06-18T02:35:28.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:28.867+0000 7f2334a5e640 1 -- 172.21.15.1:0/474469037 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2330101cb0 msgr2=0x7f2330113670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:35:28.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:28.867+0000 7f2334a5e640 1 --2- 172.21.15.1:0/474469037 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2330101cb0 0x7f2330113670 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f23180043d0 tx=0x7f2318009da0 comp rx=0 tx=0).stop 2024-06-18T02:35:28.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:28.867+0000 7f2334a5e640 1 -- 172.21.15.1:0/474469037 shutdown_connections 2024-06-18T02:35:28.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:28.867+0000 7f2334a5e640 1 --2- 172.21.15.1:0/474469037 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f231c065430 0x7f231c0678e0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:28.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:28.867+0000 7f2334a5e640 1 --2- 172.21.15.1:0/474469037 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2330101cb0 0x7f2330113670 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:28.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:28.867+0000 7f2334a5e640 1 --2- 172.21.15.1:0/474469037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2330101310 0x7f2330113130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:28.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:28.867+0000 7f2334a5e640 1 -- 172.21.15.1:0/474469037 >> 172.21.15.1:0/474469037 conn(0x7f23300f6d90 msgr2=0x7f23300f7df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:35:28.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:28.867+0000 7f2334a5e640 1 -- 172.21.15.1:0/474469037 shutdown_connections 2024-06-18T02:35:28.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:28.867+0000 7f2334a5e640 1 -- 172.21.15.1:0/474469037 wait complete. 2024-06-18T02:35:29.520 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-18T02:34:11.007633+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-06-18T02:35:29.521 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-06-18T02:35:29.784 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:35:29.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:29 smithi028 ceph-mon[35823]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:35:29.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:29 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/474469037' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-18T02:35:30.043 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:29 smithi001 ceph-mon[29099]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:35:30.043 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:29 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/474469037' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-18T02:35:31.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:31.605+0000 7f5ff2531640 1 -- 172.21.15.1:0/497627050 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5fec064d40 msgr2=0x7f5fec065110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:35:31.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:31.605+0000 7f5ff2531640 1 --2- 172.21.15.1:0/497627050 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5fec064d40 0x7f5fec065110 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc0048d0 tx=0x7f5fdc009da0 comp rx=0 tx=0).stop 2024-06-18T02:35:31.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:31.605+0000 7f5ff2531640 1 -- 172.21.15.1:0/497627050 shutdown_connections 2024-06-18T02:35:31.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:31.605+0000 7f5ff2531640 1 --2- 172.21.15.1:0/497627050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fec065650 0x7f5fec108860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:31.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:31.605+0000 7f5ff2531640 1 --2- 172.21.15.1:0/497627050 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5fec064d40 0x7f5fec065110 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:31.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:31.605+0000 7f5ff2531640 1 -- 172.21.15.1:0/497627050 >> 172.21.15.1:0/497627050 conn(0x7f5fec072110 msgr2=0x7f5fec072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:35:31.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:31.605+0000 7f5ff2531640 1 -- 172.21.15.1:0/497627050 shutdown_connections 2024-06-18T02:35:31.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:31.605+0000 7f5ff2531640 1 -- 172.21.15.1:0/497627050 wait complete. 2024-06-18T02:35:31.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:31.606+0000 7f5ff2531640 1 Processor -- start 2024-06-18T02:35:31.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:31.606+0000 7f5ff2531640 1 -- start start 2024-06-18T02:35:31.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:31.606+0000 7f5ff2531640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fec064d40 0x7f5fec115290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:35:31.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:31.606+0000 7f5ff2531640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5fec065650 0x7f5fec1157d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:35:31.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:31.606+0000 7f5ff2531640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fec115dd0 con 0x7f5fec065650 2024-06-18T02:35:31.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:31.606+0000 7f5ff2531640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fec10f210 con 0x7f5fec064d40 2024-06-18T02:35:31.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:31.606+0000 7f5fe3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5fec065650 0x7f5fec1157d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:35:31.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:31.606+0000 7f5fe3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5fec065650 0x7f5fec1157d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43422/0 (socket says 172.21.15.1:43422) 2024-06-18T02:35:31.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:31.606+0000 7f5fe3fff640 1 -- 172.21.15.1:0/1487365620 learned_addr learned my addr 172.21.15.1:0/1487365620 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:35:31.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:31.606+0000 7f5febfff640 1 --2- 172.21.15.1:0/1487365620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fec064d40 0x7f5fec115290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:35:31.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:31.606+0000 7f5fe3fff640 1 -- 172.21.15.1:0/1487365620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fec064d40 msgr2=0x7f5fec115290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:35:31.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:31.606+0000 7f5fe3fff640 1 --2- 172.21.15.1:0/1487365620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fec064d40 0x7f5fec115290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:31.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:31.606+0000 7f5fe3fff640 1 -- 172.21.15.1:0/1487365620 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fdc005040 con 0x7f5fec065650 2024-06-18T02:35:31.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:31.606+0000 7f5febfff640 1 --2- 172.21.15.1:0/1487365620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fec064d40 0x7f5fec115290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-18T02:35:31.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:31.607+0000 7f5fe3fff640 1 --2- 172.21.15.1:0/1487365620 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5fec065650 0x7f5fec1157d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5fd4002a50 tx=0x7f5fd4004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:35:31.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:31.607+0000 7f5fe97fa640 1 -- 172.21.15.1:0/1487365620 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5fd4007c30 con 0x7f5fec065650 2024-06-18T02:35:31.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:31.607+0000 7f5ff2531640 1 -- 172.21.15.1:0/1487365620 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fec10f4f0 con 0x7f5fec065650 2024-06-18T02:35:31.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:31.607+0000 7f5fe97fa640 1 -- 172.21.15.1:0/1487365620 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5fd400f040 con 0x7f5fec065650 2024-06-18T02:35:31.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:31.607+0000 7f5fe97fa640 1 -- 172.21.15.1:0/1487365620 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5fd400db50 con 0x7f5fec065650 2024-06-18T02:35:31.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:31.607+0000 7f5ff2531640 1 -- 172.21.15.1:0/1487365620 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fec10fa10 con 0x7f5fec065650 2024-06-18T02:35:31.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:31.608+0000 7f5ff2531640 1 -- 172.21.15.1:0/1487365620 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fec0fb840 con 0x7f5fec065650 2024-06-18T02:35:31.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:31.608+0000 7f5fe97fa640 1 -- 172.21.15.1:0/1487365620 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7f5fd400dcf0 con 0x7f5fec065650 2024-06-18T02:35:31.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:31.608+0000 7f5fe97fa640 1 --2- 172.21.15.1:0/1487365620 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f5fd8065430 0x7f5fd80678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:35:31.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:31.609+0000 7f5fe97fa640 1 -- 172.21.15.1:0/1487365620 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f5fd4081d30 con 0x7f5fec065650 2024-06-18T02:35:31.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:31.609+0000 7f5febfff640 1 --2- 172.21.15.1:0/1487365620 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f5fd8065430 0x7f5fd80678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:35:31.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:31.609+0000 7f5febfff640 1 --2- 172.21.15.1:0/1487365620 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f5fd8065430 0x7f5fd80678e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc032430 tx=0x7f5fdc035040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:35:31.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:31.613+0000 7f5fe97fa640 1 -- 172.21.15.1:0/1487365620 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f5fd4050260 con 0x7f5fec065650 2024-06-18T02:35:31.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:31.788+0000 7f5ff2531640 1 -- 172.21.15.1:0/1487365620 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5fec110900 con 0x7f5fec065650 2024-06-18T02:35:31.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:31.788+0000 7f5fe97fa640 1 -- 172.21.15.1:0/1487365620 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v40) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5fd4053880 con 0x7f5fec065650 2024-06-18T02:35:31.788 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-06-18T02:35:31.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:31.790+0000 7f5ff2531640 1 -- 172.21.15.1:0/1487365620 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f5fd8065430 msgr2=0x7f5fd80678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:35:31.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:31.790+0000 7f5ff2531640 1 --2- 172.21.15.1:0/1487365620 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f5fd8065430 0x7f5fd80678e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc032430 tx=0x7f5fdc035040 comp rx=0 tx=0).stop 2024-06-18T02:35:31.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:31.790+0000 7f5ff2531640 1 -- 172.21.15.1:0/1487365620 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5fec065650 msgr2=0x7f5fec1157d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:35:31.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:31.790+0000 7f5ff2531640 1 --2- 172.21.15.1:0/1487365620 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5fec065650 0x7f5fec1157d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5fd4002a50 tx=0x7f5fd4004600 comp rx=0 tx=0).stop 2024-06-18T02:35:31.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:31.790+0000 7f5ff2531640 1 -- 172.21.15.1:0/1487365620 shutdown_connections 2024-06-18T02:35:31.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:31.790+0000 7f5ff2531640 1 --2- 172.21.15.1:0/1487365620 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f5fd8065430 0x7f5fd80678e0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:31.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:31.790+0000 7f5ff2531640 1 --2- 172.21.15.1:0/1487365620 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5fec065650 0x7f5fec1157d0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:31.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:31.790+0000 7f5ff2531640 1 --2- 172.21.15.1:0/1487365620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fec064d40 0x7f5fec115290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:31.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:31.790+0000 7f5ff2531640 1 -- 172.21.15.1:0/1487365620 >> 172.21.15.1:0/1487365620 conn(0x7f5fec072110 msgr2=0x7f5fec0f8c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:35:31.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:31.790+0000 7f5ff2531640 1 -- 172.21.15.1:0/1487365620 shutdown_connections 2024-06-18T02:35:31.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:31.790+0000 7f5ff2531640 1 -- 172.21.15.1:0/1487365620 wait complete. 2024-06-18T02:35:32.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:31 smithi001 ceph-mon[29099]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:35:32.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:31 smithi028 ceph-mon[35823]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:35:32.522 INFO:tasks.cephadm:Setting up client nodes... 2024-06-18T02:35:32.523 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-06-18T02:35:32.794 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:35:33.034 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:32 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/1487365620' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-06-18T02:35:33.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:32 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/1487365620' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-06-18T02:35:34.054 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:33 smithi001 ceph-mon[29099]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:35:34.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:33 smithi028 ceph-mon[35823]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:35:34.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:34.383+0000 7f0fc8778640 1 -- 172.21.15.1:0/702725335 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fc00fe330 msgr2=0x7f0fc00fa490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:35:34.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:34.383+0000 7f0fc8778640 1 --2- 172.21.15.1:0/702725335 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fc00fe330 0x7f0fc00fa490 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f0fb00048d0 tx=0x7f0fb0009da0 comp rx=0 tx=0).stop 2024-06-18T02:35:34.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:34.384+0000 7f0fc8778640 1 -- 172.21.15.1:0/702725335 shutdown_connections 2024-06-18T02:35:34.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:34.384+0000 7f0fc8778640 1 --2- 172.21.15.1:0/702725335 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fc00fe330 0x7f0fc00fa490 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:34.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:34.384+0000 7f0fc8778640 1 --2- 172.21.15.1:0/702725335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0fc00fd9e0 0x7f0fc00fddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:34.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:34.384+0000 7f0fc8778640 1 -- 172.21.15.1:0/702725335 >> 172.21.15.1:0/702725335 conn(0x7f0fc00f6100 msgr2=0x7f0fc00f8550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:35:34.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:34.384+0000 7f0fc8778640 1 -- 172.21.15.1:0/702725335 shutdown_connections 2024-06-18T02:35:34.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:34.384+0000 7f0fc8778640 1 -- 172.21.15.1:0/702725335 wait complete. 2024-06-18T02:35:34.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:34.384+0000 7f0fc8778640 1 Processor -- start 2024-06-18T02:35:34.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:34.384+0000 7f0fc8778640 1 -- start start 2024-06-18T02:35:34.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:34.384+0000 7f0fc8778640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fc00fd9e0 0x7f0fc010cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:35:34.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:34.384+0000 7f0fc8778640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0fc00fe330 0x7f0fc010d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:35:34.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:34.384+0000 7f0fc8778640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc010da80 con 0x7f0fc00fd9e0 2024-06-18T02:35:34.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:34.384+0000 7f0fc8778640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc010e250 con 0x7f0fc00fe330 2024-06-18T02:35:34.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:34.384+0000 7f0fc6514640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fc00fd9e0 0x7f0fc010cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:35:34.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:34.384+0000 7f0fc6514640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fc00fd9e0 0x7f0fc010cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35606/0 (socket says 172.21.15.1:35606) 2024-06-18T02:35:34.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:34.384+0000 7f0fc6514640 1 -- 172.21.15.1:0/2275943494 learned_addr learned my addr 172.21.15.1:0/2275943494 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:35:34.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:34.385+0000 7f0fc5d13640 1 --2- 172.21.15.1:0/2275943494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0fc00fe330 0x7f0fc010d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:35:34.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:34.385+0000 7f0fc6514640 1 -- 172.21.15.1:0/2275943494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0fc00fe330 msgr2=0x7f0fc010d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:35:34.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:34.385+0000 7f0fc6514640 1 --2- 172.21.15.1:0/2275943494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0fc00fe330 0x7f0fc010d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:34.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:34.385+0000 7f0fc6514640 1 -- 172.21.15.1:0/2275943494 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fb0005040 con 0x7f0fc00fd9e0 2024-06-18T02:35:34.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:34.385+0000 7f0fc6514640 1 --2- 172.21.15.1:0/2275943494 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fc00fd9e0 0x7f0fc010cef0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc002a50 tx=0x7f0fbc004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:35:34.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:34.385+0000 7f0fb6ffd640 1 -- 172.21.15.1:0/2275943494 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0fbc007c30 con 0x7f0fc00fd9e0 2024-06-18T02:35:34.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:34.385+0000 7f0fb6ffd640 1 -- 172.21.15.1:0/2275943494 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0fbc00f040 con 0x7f0fc00fd9e0 2024-06-18T02:35:34.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:34.385+0000 7f0fc8778640 1 -- 172.21.15.1:0/2275943494 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fc010e4e0 con 0x7f0fc00fd9e0 2024-06-18T02:35:34.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:34.385+0000 7f0fb6ffd640 1 -- 172.21.15.1:0/2275943494 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0fbc00dbe0 con 0x7f0fc00fd9e0 2024-06-18T02:35:34.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:34.385+0000 7f0fc8778640 1 -- 172.21.15.1:0/2275943494 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fc010ea80 con 0x7f0fc00fd9e0 2024-06-18T02:35:34.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:34.386+0000 7f0fc8778640 1 -- 172.21.15.1:0/2275943494 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fc00f9dc0 con 0x7f0fc00fd9e0 2024-06-18T02:35:34.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:34.387+0000 7f0fb6ffd640 1 -- 172.21.15.1:0/2275943494 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7f0fbc00dd40 con 0x7f0fc00fd9e0 2024-06-18T02:35:34.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:34.387+0000 7f0fb6ffd640 1 --2- 172.21.15.1:0/2275943494 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f0fac065430 0x7f0fac0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:35:34.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:34.387+0000 7f0fb6ffd640 1 -- 172.21.15.1:0/2275943494 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f0fbc0823a0 con 0x7f0fc00fd9e0 2024-06-18T02:35:34.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:34.387+0000 7f0fc5d13640 1 --2- 172.21.15.1:0/2275943494 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f0fac065430 0x7f0fac0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:35:34.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:34.388+0000 7f0fc5d13640 1 --2- 172.21.15.1:0/2275943494 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f0fac065430 0x7f0fac0678e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0fc010cca0 tx=0x7f0fb0035040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:35:34.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:34.392+0000 7f0fb6ffd640 1 -- 172.21.15.1:0/2275943494 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f0fbc0508d0 con 0x7f0fc00fd9e0 2024-06-18T02:35:34.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:34.630+0000 7f0fc8778640 1 -- 172.21.15.1:0/2275943494 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f0fc004b480 con 0x7f0fc00fd9e0 2024-06-18T02:35:34.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:34.634+0000 7f0fb6ffd640 1 -- 172.21.15.1:0/2275943494 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v14) v1 ==== 170+0+59 (secure 0 0 0) 0x7f0fbc053ef0 con 0x7f0fc00fd9e0 2024-06-18T02:35:34.634 INFO:teuthology.orchestra.run.smithi001.stdout:[client.0] 2024-06-18T02:35:34.634 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQB28nBmauenJRAARRe2MUhzJ1CDZP0DiqR42A== 2024-06-18T02:35:34.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:34.636+0000 7f0fc8778640 1 -- 172.21.15.1:0/2275943494 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f0fac065430 msgr2=0x7f0fac0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:35:34.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:34.636+0000 7f0fc8778640 1 --2- 172.21.15.1:0/2275943494 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f0fac065430 0x7f0fac0678e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0fc010cca0 tx=0x7f0fb0035040 comp rx=0 tx=0).stop 2024-06-18T02:35:34.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:34.636+0000 7f0fc8778640 1 -- 172.21.15.1:0/2275943494 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fc00fd9e0 msgr2=0x7f0fc010cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:35:34.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:34.636+0000 7f0fc8778640 1 --2- 172.21.15.1:0/2275943494 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fc00fd9e0 0x7f0fc010cef0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc002a50 tx=0x7f0fbc004600 comp rx=0 tx=0).stop 2024-06-18T02:35:34.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:34.636+0000 7f0fc8778640 1 -- 172.21.15.1:0/2275943494 shutdown_connections 2024-06-18T02:35:34.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:34.636+0000 7f0fc8778640 1 --2- 172.21.15.1:0/2275943494 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f0fac065430 0x7f0fac0678e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:34.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:34.636+0000 7f0fc8778640 1 --2- 172.21.15.1:0/2275943494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0fc00fe330 0x7f0fc010d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:34.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:34.636+0000 7f0fc8778640 1 --2- 172.21.15.1:0/2275943494 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fc00fd9e0 0x7f0fc010cef0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:34.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:34.636+0000 7f0fc8778640 1 -- 172.21.15.1:0/2275943494 >> 172.21.15.1:0/2275943494 conn(0x7f0fc00f6100 msgr2=0x7f0fc00f7780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:35:34.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:34.636+0000 7f0fc8778640 1 -- 172.21.15.1:0/2275943494 shutdown_connections 2024-06-18T02:35:34.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:34.636+0000 7f0fc8778640 1 -- 172.21.15.1:0/2275943494 wait complete. 2024-06-18T02:35:35.003 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:34 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/2275943494' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-18T02:35:35.004 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:34 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/2275943494' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-18T02:35:35.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:34 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/2275943494' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-18T02:35:35.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:34 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/2275943494' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-18T02:35:35.436 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-06-18T02:35:35.436 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-06-18T02:35:35.436 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-06-18T02:35:35.479 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-06-18T02:35:35.755 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi028/config 2024-06-18T02:35:36.127 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:35 smithi001 ceph-mon[29099]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:35:36.133 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:35 smithi028 ceph-mon[35823]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:35:37.099 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:36 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:35:37.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:36 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:35:37.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:37.437+0000 7f4580963640 1 -- 172.21.15.28:0/248362630 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f457c0663f0 msgr2=0x7f457c0fef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:35:37.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:37.437+0000 7f4580963640 1 --2- 172.21.15.28:0/248362630 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f457c0663f0 0x7f457c0fef70 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f456c0048d0 tx=0x7f456c009da0 comp rx=0 tx=0).stop 2024-06-18T02:35:37.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:37.438+0000 7f4580963640 1 -- 172.21.15.28:0/248362630 shutdown_connections 2024-06-18T02:35:37.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:37.438+0000 7f4580963640 1 --2- 172.21.15.28:0/248362630 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f457c0ff4b0 0x7f457c101890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:37.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:37.438+0000 7f4580963640 1 --2- 172.21.15.28:0/248362630 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f457c0663f0 0x7f457c0fef70 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:37.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:37.438+0000 7f4580963640 1 -- 172.21.15.28:0/248362630 >> 172.21.15.28:0/248362630 conn(0x7f457c0f69a0 msgr2=0x7f457c0f8dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:35:37.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:37.438+0000 7f4580963640 1 -- 172.21.15.28:0/248362630 shutdown_connections 2024-06-18T02:35:37.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:37.438+0000 7f4580963640 1 -- 172.21.15.28:0/248362630 wait complete. 2024-06-18T02:35:37.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:37.438+0000 7f4580963640 1 Processor -- start 2024-06-18T02:35:37.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:37.438+0000 7f4580963640 1 -- start start 2024-06-18T02:35:37.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:37.438+0000 7f4580963640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f457c0663f0 0x7f457c10cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:35:37.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:37.438+0000 7f4580963640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f457c0ff4b0 0x7f457c10d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:35:37.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:37.438+0000 7f4580963640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f457c10da80 con 0x7f457c0663f0 2024-06-18T02:35:37.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:37.438+0000 7f4580963640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f457c10e2a0 con 0x7f457c0ff4b0 2024-06-18T02:35:37.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:37.438+0000 7f4573fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f457c0ff4b0 0x7f457c10d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:35:37.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:37.439+0000 7f4573fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f457c0ff4b0 0x7f457c10d430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34142/0 (socket says 172.21.15.28:34142) 2024-06-18T02:35:37.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:37.439+0000 7f4573fff640 1 -- 172.21.15.28:0/139408886 learned_addr learned my addr 172.21.15.28:0/139408886 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-06-18T02:35:37.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:37.439+0000 7f457a59c640 1 --2- 172.21.15.28:0/139408886 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f457c0663f0 0x7f457c10cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:35:37.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:37.439+0000 7f4573fff640 1 -- 172.21.15.28:0/139408886 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f457c0663f0 msgr2=0x7f457c10cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:35:37.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:37.439+0000 7f4573fff640 1 --2- 172.21.15.28:0/139408886 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f457c0663f0 0x7f457c10cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:37.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:37.439+0000 7f4573fff640 1 -- 172.21.15.28:0/139408886 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f456c005040 con 0x7f457c0ff4b0 2024-06-18T02:35:37.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:37.439+0000 7f4573fff640 1 --2- 172.21.15.28:0/139408886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f457c0ff4b0 0x7f457c10d430 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f4564002840 tx=0x7f4564004570 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:35:37.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:37.439+0000 7f4572ffd640 1 -- 172.21.15.28:0/139408886 <== mon.1 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4564007c30 con 0x7f457c0ff4b0 2024-06-18T02:35:37.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:37.439+0000 7f4580963640 1 -- 172.21.15.28:0/139408886 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f457c10e530 con 0x7f457c0ff4b0 2024-06-18T02:35:37.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:37.439+0000 7f4572ffd640 1 -- 172.21.15.28:0/139408886 <== mon.1 v2:172.21.15.28:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f456400f040 con 0x7f457c0ff4b0 2024-06-18T02:35:37.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:37.439+0000 7f4572ffd640 1 -- 172.21.15.28:0/139408886 <== mon.1 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f456400da30 con 0x7f457c0ff4b0 2024-06-18T02:35:37.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:37.439+0000 7f4580963640 1 -- 172.21.15.28:0/139408886 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f457c10ead0 con 0x7f457c0ff4b0 2024-06-18T02:35:37.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:37.440+0000 7f4580963640 1 -- 172.21.15.28:0/139408886 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f457c0fa640 con 0x7f457c0ff4b0 2024-06-18T02:35:37.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:37.441+0000 7f4572ffd640 1 -- 172.21.15.28:0/139408886 <== mon.1 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7f45640096a0 con 0x7f457c0ff4b0 2024-06-18T02:35:37.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:37.441+0000 7f4572ffd640 1 --2- 172.21.15.28:0/139408886 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f4568065430 0x7f45680678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:35:37.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:37.441+0000 7f4572ffd640 1 -- 172.21.15.28:0/139408886 <== mon.1 v2:172.21.15.28:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f4564082290 con 0x7f457c0ff4b0 2024-06-18T02:35:37.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:37.442+0000 7f457a59c640 1 --2- 172.21.15.28:0/139408886 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f4568065430 0x7f45680678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:35:37.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:37.442+0000 7f457a59c640 1 --2- 172.21.15.28:0/139408886 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f4568065430 0x7f45680678e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f456c000c00 tx=0x7f456c035040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:35:37.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:37.446+0000 7f4572ffd640 1 -- 172.21.15.28:0/139408886 <== mon.1 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f4564050740 con 0x7f457c0ff4b0 2024-06-18T02:35:37.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:37.682+0000 7f4580963640 1 -- 172.21.15.28:0/139408886 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f457c10ed90 con 0x7f457c0ff4b0 2024-06-18T02:35:37.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:37.687+0000 7f4572ffd640 1 -- 172.21.15.28:0/139408886 <== mon.1 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v15) v1 ==== 170+0+59 (secure 0 0 0) 0x7f4564053d60 con 0x7f457c0ff4b0 2024-06-18T02:35:37.687 INFO:teuthology.orchestra.run.smithi028.stdout:[client.1] 2024-06-18T02:35:37.687 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQB58nBm1XjIKBAABWTHflaAbOMHlFj/8Fw7sw== 2024-06-18T02:35:37.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:37.689+0000 7f4580963640 1 -- 172.21.15.28:0/139408886 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f4568065430 msgr2=0x7f45680678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:35:37.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:37.689+0000 7f4580963640 1 --2- 172.21.15.28:0/139408886 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f4568065430 0x7f45680678e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f456c000c00 tx=0x7f456c035040 comp rx=0 tx=0).stop 2024-06-18T02:35:37.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:37.689+0000 7f4580963640 1 -- 172.21.15.28:0/139408886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f457c0ff4b0 msgr2=0x7f457c10d430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:35:37.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:37.689+0000 7f4580963640 1 --2- 172.21.15.28:0/139408886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f457c0ff4b0 0x7f457c10d430 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f4564002840 tx=0x7f4564004570 comp rx=0 tx=0).stop 2024-06-18T02:35:37.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:37.689+0000 7f4580963640 1 -- 172.21.15.28:0/139408886 shutdown_connections 2024-06-18T02:35:37.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:37.689+0000 7f4580963640 1 --2- 172.21.15.28:0/139408886 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f4568065430 0x7f45680678e0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:37.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:37.689+0000 7f4580963640 1 --2- 172.21.15.28:0/139408886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f457c0ff4b0 0x7f457c10d430 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:37.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:37.689+0000 7f4580963640 1 --2- 172.21.15.28:0/139408886 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f457c0663f0 0x7f457c10cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:37.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:37.689+0000 7f4580963640 1 -- 172.21.15.28:0/139408886 >> 172.21.15.28:0/139408886 conn(0x7f457c0f69a0 msgr2=0x7f457c0f6dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:35:37.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:37.689+0000 7f4580963640 1 -- 172.21.15.28:0/139408886 shutdown_connections 2024-06-18T02:35:37.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-06-18T02:35:37.689+0000 7f4580963640 1 -- 172.21.15.28:0/139408886 wait complete. 2024-06-18T02:35:37.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:37 smithi028 ceph-mon[35823]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:35:37.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:37 smithi028 ceph-mon[35823]: from='client.? 172.21.15.28:0/139408886' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-18T02:35:37.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:37 smithi028 ceph-mon[35823]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-18T02:35:37.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:37 smithi028 ceph-mon[35823]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-18T02:35:38.127 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:37 smithi001 ceph-mon[29099]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:35:38.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:37 smithi001 ceph-mon[29099]: from='client.? 172.21.15.28:0/139408886' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-18T02:35:38.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:37 smithi001 ceph-mon[29099]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-18T02:35:38.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:37 smithi001 ceph-mon[29099]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-18T02:35:38.249 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-06-18T02:35:38.249 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-06-18T02:35:38.249 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-06-18T02:35:38.289 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-06-18T02:35:38.289 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-06-18T02:35:38.289 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph mgr dump --format=json 2024-06-18T02:35:38.563 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:35:39.992 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:39 smithi001 ceph-mon[29099]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:35:40.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:40.220+0000 7f05a604d640 1 -- 172.21.15.1:0/3856487501 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f05a0101310 msgr2=0x7f05a01016e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:35:40.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:40.220+0000 7f05a604d640 1 --2- 172.21.15.1:0/3856487501 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f05a0101310 0x7f05a01016e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f05900048d0 tx=0x7f0590009da0 comp rx=0 tx=0).stop 2024-06-18T02:35:40.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:40.221+0000 7f05a604d640 1 -- 172.21.15.1:0/3856487501 shutdown_connections 2024-06-18T02:35:40.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:40.221+0000 7f05a604d640 1 --2- 172.21.15.1:0/3856487501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05a0101cb0 0x7f05a00fb230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:40.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:40.221+0000 7f05a604d640 1 --2- 172.21.15.1:0/3856487501 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f05a0101310 0x7f05a01016e0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:40.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:40.221+0000 7f05a604d640 1 -- 172.21.15.1:0/3856487501 >> 172.21.15.1:0/3856487501 conn(0x7f05a00f6d90 msgr2=0x7f05a00f91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:35:40.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:40.221+0000 7f05a604d640 1 -- 172.21.15.1:0/3856487501 shutdown_connections 2024-06-18T02:35:40.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:40.221+0000 7f05a604d640 1 -- 172.21.15.1:0/3856487501 wait complete. 2024-06-18T02:35:40.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:40.221+0000 7f05a604d640 1 Processor -- start 2024-06-18T02:35:40.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:40.221+0000 7f05a604d640 1 -- start start 2024-06-18T02:35:40.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:40.222+0000 7f05a604d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05a0101310 0x7f05a0113190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:35:40.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:40.222+0000 7f05a604d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f05a0101cb0 0x7f05a01136d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:35:40.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:40.222+0000 7f05a604d640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a0113cd0 con 0x7f05a0101cb0 2024-06-18T02:35:40.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:40.222+0000 7f05a604d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a010d110 con 0x7f05a0101310 2024-06-18T02:35:40.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:40.222+0000 7f059effd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f05a0101cb0 0x7f05a01136d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:35:40.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:40.222+0000 7f059effd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f05a0101cb0 0x7f05a01136d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35628/0 (socket says 172.21.15.1:35628) 2024-06-18T02:35:40.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:40.222+0000 7f059effd640 1 -- 172.21.15.1:0/334468462 learned_addr learned my addr 172.21.15.1:0/334468462 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:35:40.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:40.222+0000 7f059f7fe640 1 --2- 172.21.15.1:0/334468462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05a0101310 0x7f05a0113190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:35:40.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:40.222+0000 7f059effd640 1 -- 172.21.15.1:0/334468462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05a0101310 msgr2=0x7f05a0113190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:35:40.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:40.222+0000 7f059effd640 1 --2- 172.21.15.1:0/334468462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05a0101310 0x7f05a0113190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:40.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:40.222+0000 7f059effd640 1 -- 172.21.15.1:0/334468462 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0590005040 con 0x7f05a0101cb0 2024-06-18T02:35:40.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:40.222+0000 7f059f7fe640 1 --2- 172.21.15.1:0/334468462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05a0101310 0x7f05a0113190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-18T02:35:40.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:40.222+0000 7f059effd640 1 --2- 172.21.15.1:0/334468462 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f05a0101cb0 0x7f05a01136d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f059400eb70 tx=0x7f059400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:35:40.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:40.222+0000 7f0587fff640 1 -- 172.21.15.1:0/334468462 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0594007c30 con 0x7f05a0101cb0 2024-06-18T02:35:40.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:40.222+0000 7f0587fff640 1 -- 172.21.15.1:0/334468462 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f059400f040 con 0x7f05a0101cb0 2024-06-18T02:35:40.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:40.222+0000 7f0587fff640 1 -- 172.21.15.1:0/334468462 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0594009680 con 0x7f05a0101cb0 2024-06-18T02:35:40.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:40.222+0000 7f05a604d640 1 -- 172.21.15.1:0/334468462 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05a010d390 con 0x7f05a0101cb0 2024-06-18T02:35:40.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:40.223+0000 7f05a604d640 1 -- 172.21.15.1:0/334468462 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05a010d8b0 con 0x7f05a0101cb0 2024-06-18T02:35:40.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:40.224+0000 7f05a604d640 1 -- 172.21.15.1:0/334468462 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05a00fa9f0 con 0x7f05a0101cb0 2024-06-18T02:35:40.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:40.224+0000 7f0587fff640 1 -- 172.21.15.1:0/334468462 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7f059400cd40 con 0x7f05a0101cb0 2024-06-18T02:35:40.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:40.224+0000 7f0587fff640 1 --2- 172.21.15.1:0/334468462 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f0588065430 0x7f05880678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:35:40.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:40.224+0000 7f0587fff640 1 -- 172.21.15.1:0/334468462 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f05940820f0 con 0x7f05a0101cb0 2024-06-18T02:35:40.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:40.225+0000 7f059f7fe640 1 --2- 172.21.15.1:0/334468462 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f0588065430 0x7f05880678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:35:40.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:40.225+0000 7f059f7fe640 1 --2- 172.21.15.1:0/334468462 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f0588065430 0x7f05880678e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0590009f80 tx=0x7f0590035040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:35:40.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:40.229+0000 7f0587fff640 1 -- 172.21.15.1:0/334468462 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f059404ddf0 con 0x7f05a0101cb0 2024-06-18T02:35:40.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:39 smithi028 ceph-mon[35823]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:35:40.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:40.447+0000 7f05a604d640 1 -- 172.21.15.1:0/334468462 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f05a004b480 con 0x7f05a0101cb0 2024-06-18T02:35:40.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:40.450+0000 7f0587fff640 1 -- 172.21.15.1:0/334468462 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v19) v1 ==== 74+0+161434 (secure 0 0 0) 0x7f059404a6b0 con 0x7f05a0101cb0 2024-06-18T02:35:40.453 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:35:40.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:40.452+0000 7f05a604d640 1 -- 172.21.15.1:0/334468462 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f0588065430 msgr2=0x7f05880678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:35:40.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:40.452+0000 7f05a604d640 1 --2- 172.21.15.1:0/334468462 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f0588065430 0x7f05880678e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0590009f80 tx=0x7f0590035040 comp rx=0 tx=0).stop 2024-06-18T02:35:40.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:40.452+0000 7f05a604d640 1 -- 172.21.15.1:0/334468462 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f05a0101cb0 msgr2=0x7f05a01136d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:35:40.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:40.452+0000 7f05a604d640 1 --2- 172.21.15.1:0/334468462 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f05a0101cb0 0x7f05a01136d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f059400eb70 tx=0x7f059400b040 comp rx=0 tx=0).stop 2024-06-18T02:35:40.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:40.452+0000 7f05a604d640 1 -- 172.21.15.1:0/334468462 shutdown_connections 2024-06-18T02:35:40.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:40.452+0000 7f05a604d640 1 --2- 172.21.15.1:0/334468462 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f0588065430 0x7f05880678e0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:40.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:40.452+0000 7f05a604d640 1 --2- 172.21.15.1:0/334468462 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f05a0101cb0 0x7f05a01136d0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:40.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:40.452+0000 7f05a604d640 1 --2- 172.21.15.1:0/334468462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05a0101310 0x7f05a0113190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:40.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:40.452+0000 7f05a604d640 1 -- 172.21.15.1:0/334468462 >> 172.21.15.1:0/334468462 conn(0x7f05a00f6d90 msgr2=0x7f05a00f8450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:35:40.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:40.452+0000 7f05a604d640 1 -- 172.21.15.1:0/334468462 shutdown_connections 2024-06-18T02:35:40.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:40.452+0000 7f05a604d640 1 -- 172.21.15.1:0/334468462 wait complete. 2024-06-18T02:35:41.127 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:40 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/334468462' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-06-18T02:35:41.197 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":19,"flags":0,"active_gid":14216,"active_name":"smithi001.teoawz","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":729353917},{"type":"v1","addr":"172.21.15.1:6801","nonce":729353917}]},"active_addr":"172.21.15.1:6801/729353917","active_change":"2024-06-18T02:32:06.496996+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":14234,"name":"smithi028.xnuopx","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":false,"error_string":"No module named 'kubernetes.client.models.v1_event'","module_options":{}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":false,"error_string":"No module named 'kubernetes.client.models.v1_event'","module_options":{}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.1:8443/","prometheus":"http://172.21.15.1:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.1:0","nonce":2018035679}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.1:0","nonce":1927947308}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.1:0","nonce":3017630763}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.1:0","nonce":473963939}]}]} 2024-06-18T02:35:41.201 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-06-18T02:35:41.201 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-06-18T02:35:41.201 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph osd dump --format=json 2024-06-18T02:35:41.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:40 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/334468462' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-06-18T02:35:41.469 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:35:41.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:41 smithi001 ceph-mon[29099]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:35:42.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:41 smithi028 ceph-mon[35823]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:35:43.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:43.143+0000 7fa1bc542640 1 -- 172.21.15.1:0/876571963 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1b41022a0 msgr2=0x7fa1b4102670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:35:43.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:43.143+0000 7fa1bc542640 1 --2- 172.21.15.1:0/876571963 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1b41022a0 0x7fa1b4102670 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fa1b00048d0 tx=0x7fa1b0009da0 comp rx=0 tx=0).stop 2024-06-18T02:35:43.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:43.143+0000 7fa1bc542640 1 -- 172.21.15.1:0/876571963 shutdown_connections 2024-06-18T02:35:43.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:43.143+0000 7fa1bc542640 1 --2- 172.21.15.1:0/876571963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1b4102c40 0x7fa1b40fc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:43.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:43.143+0000 7fa1bc542640 1 --2- 172.21.15.1:0/876571963 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1b41022a0 0x7fa1b4102670 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:43.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:43.143+0000 7fa1bc542640 1 -- 172.21.15.1:0/876571963 >> 172.21.15.1:0/876571963 conn(0x7fa1b40f7e50 msgr2=0x7fa1b40fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:35:43.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:43.143+0000 7fa1bc542640 1 -- 172.21.15.1:0/876571963 shutdown_connections 2024-06-18T02:35:43.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:43.143+0000 7fa1bc542640 1 -- 172.21.15.1:0/876571963 wait complete. 2024-06-18T02:35:43.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:43.143+0000 7fa1bc542640 1 Processor -- start 2024-06-18T02:35:43.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:43.143+0000 7fa1bc542640 1 -- start start 2024-06-18T02:35:43.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:43.144+0000 7fa1bc542640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1b41022a0 0x7fa1b4115220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:35:43.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:43.144+0000 7fa1bc542640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1b4102c40 0x7fa1b4115760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:35:43.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:43.144+0000 7fa1bc542640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1b4115d60 con 0x7fa1b41022a0 2024-06-18T02:35:43.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:43.144+0000 7fa1bc542640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1b410f1a0 con 0x7fa1b4102c40 2024-06-18T02:35:43.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:43.144+0000 7fa1ba2de640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1b41022a0 0x7fa1b4115220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:35:43.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:43.144+0000 7fa1ba2de640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1b41022a0 0x7fa1b4115220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35632/0 (socket says 172.21.15.1:35632) 2024-06-18T02:35:43.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:43.144+0000 7fa1ba2de640 1 -- 172.21.15.1:0/2187291769 learned_addr learned my addr 172.21.15.1:0/2187291769 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:35:43.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:43.144+0000 7fa1b9add640 1 --2- 172.21.15.1:0/2187291769 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1b4102c40 0x7fa1b4115760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:35:43.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:43.144+0000 7fa1ba2de640 1 -- 172.21.15.1:0/2187291769 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1b4102c40 msgr2=0x7fa1b4115760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:35:43.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:43.144+0000 7fa1ba2de640 1 --2- 172.21.15.1:0/2187291769 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1b4102c40 0x7fa1b4115760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:43.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:43.144+0000 7fa1ba2de640 1 -- 172.21.15.1:0/2187291769 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1b0005040 con 0x7fa1b41022a0 2024-06-18T02:35:43.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:43.144+0000 7fa1ba2de640 1 --2- 172.21.15.1:0/2187291769 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1b41022a0 0x7fa1b4115220 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fa1b00043d0 tx=0x7fa1b0009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:35:43.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:43.145+0000 7fa1aaffd640 1 -- 172.21.15.1:0/2187291769 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa1b0038070 con 0x7fa1b41022a0 2024-06-18T02:35:43.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:43.145+0000 7fa1aaffd640 1 -- 172.21.15.1:0/2187291769 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa1b0032970 con 0x7fa1b41022a0 2024-06-18T02:35:43.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:43.145+0000 7fa1bc542640 1 -- 172.21.15.1:0/2187291769 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1b410f420 con 0x7fa1b41022a0 2024-06-18T02:35:43.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:43.145+0000 7fa1aaffd640 1 -- 172.21.15.1:0/2187291769 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa1b003da60 con 0x7fa1b41022a0 2024-06-18T02:35:43.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:43.145+0000 7fa1bc542640 1 -- 172.21.15.1:0/2187291769 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1b410f960 con 0x7fa1b41022a0 2024-06-18T02:35:43.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:43.146+0000 7fa1bc542640 1 -- 172.21.15.1:0/2187291769 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1b40fda10 con 0x7fa1b41022a0 2024-06-18T02:35:43.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:43.146+0000 7fa1aaffd640 1 -- 172.21.15.1:0/2187291769 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7fa1b0030020 con 0x7fa1b41022a0 2024-06-18T02:35:43.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:43.147+0000 7fa1aaffd640 1 --2- 172.21.15.1:0/2187291769 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fa1a0065430 0x7fa1a00678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:35:43.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:43.147+0000 7fa1aaffd640 1 -- 172.21.15.1:0/2187291769 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fa1b00a25c0 con 0x7fa1b41022a0 2024-06-18T02:35:43.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:43.147+0000 7fa1b9add640 1 --2- 172.21.15.1:0/2187291769 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fa1a0065430 0x7fa1a00678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:35:43.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:43.147+0000 7fa1b9add640 1 --2- 172.21.15.1:0/2187291769 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fa1a0065430 0x7fa1a00678e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa1b4110ae0 tx=0x7fa1a400b040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:35:43.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:43.151+0000 7fa1aaffd640 1 -- 172.21.15.1:0/2187291769 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fa1b0070a70 con 0x7fa1b41022a0 2024-06-18T02:35:43.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:43.327+0000 7fa1bc542640 1 -- 172.21.15.1:0/2187291769 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa1b40fba90 con 0x7fa1b41022a0 2024-06-18T02:35:43.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:43.328+0000 7fa1aaffd640 1 -- 172.21.15.1:0/2187291769 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+10965 (secure 0 0 0) 0x7fa1b0074090 con 0x7fa1b41022a0 2024-06-18T02:35:43.329 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:35:43.329 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":40,"fsid":"7eefb63e-2d1a-11ef-bc9d-c7b262605968","created":"2024-06-18T02:29:58.467010+0000","modified":"2024-06-18T02:35:22.963059+0000","last_up_change":"2024-06-18T02:35:21.961141+0000","last_in_change":"2024-06-18T02:35:03.501632+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-18T02:34:11.007633+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"2fc85253-858e-496d-a262-061526504dc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":797495043},{"type":"v1","addr":"172.21.15.1:6803","nonce":797495043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":797495043},{"type":"v1","addr":"172.21.15.1:6805","nonce":797495043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":797495043},{"type":"v1","addr":"172.21.15.1:6809","nonce":797495043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":797495043},{"type":"v1","addr":"172.21.15.1:6807","nonce":797495043}]},"public_addr":"172.21.15.1:6803/797495043","cluster_addr":"172.21.15.1:6805/797495043","heartbeat_back_addr":"172.21.15.1:6809/797495043","heartbeat_front_addr":"172.21.15.1:6807/797495043","state":["exists","up"]},{"osd":1,"uuid":"f9bb7834-4fa4-40b1-9b63-bd1a69a4e082","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":2182416687},{"type":"v1","addr":"172.21.15.1:6811","nonce":2182416687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":2182416687},{"type":"v1","addr":"172.21.15.1:6813","nonce":2182416687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":2182416687},{"type":"v1","addr":"172.21.15.1:6817","nonce":2182416687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":2182416687},{"type":"v1","addr":"172.21.15.1:6815","nonce":2182416687}]},"public_addr":"172.21.15.1:6811/2182416687","cluster_addr":"172.21.15.1:6813/2182416687","heartbeat_back_addr":"172.21.15.1:6817/2182416687","heartbeat_front_addr":"172.21.15.1:6815/2182416687","state":["exists","up"]},{"osd":2,"uuid":"89223bec-bba5-4744-960e-5396f04ddba1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":3682942580},{"type":"v1","addr":"172.21.15.1:6819","nonce":3682942580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3682942580},{"type":"v1","addr":"172.21.15.1:6821","nonce":3682942580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3682942580},{"type":"v1","addr":"172.21.15.1:6825","nonce":3682942580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":3682942580},{"type":"v1","addr":"172.21.15.1:6823","nonce":3682942580}]},"public_addr":"172.21.15.1:6819/3682942580","cluster_addr":"172.21.15.1:6821/3682942580","heartbeat_back_addr":"172.21.15.1:6825/3682942580","heartbeat_front_addr":"172.21.15.1:6823/3682942580","state":["exists","up"]},{"osd":3,"uuid":"33dd638d-10d2-432a-92ac-74b68b4a8311","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":664150046},{"type":"v1","addr":"172.21.15.28:6801","nonce":664150046}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":664150046},{"type":"v1","addr":"172.21.15.28:6803","nonce":664150046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":664150046},{"type":"v1","addr":"172.21.15.28:6807","nonce":664150046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":664150046},{"type":"v1","addr":"172.21.15.28:6805","nonce":664150046}]},"public_addr":"172.21.15.28:6801/664150046","cluster_addr":"172.21.15.28:6803/664150046","heartbeat_back_addr":"172.21.15.28:6807/664150046","heartbeat_front_addr":"172.21.15.28:6805/664150046","state":["exists","up"]},{"osd":4,"uuid":"2f4a6526-2f3e-417f-b02c-cf242e7eac07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":276884153},{"type":"v1","addr":"172.21.15.28:6809","nonce":276884153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":276884153},{"type":"v1","addr":"172.21.15.28:6811","nonce":276884153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":276884153},{"type":"v1","addr":"172.21.15.28:6815","nonce":276884153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":276884153},{"type":"v1","addr":"172.21.15.28:6813","nonce":276884153}]},"public_addr":"172.21.15.28:6809/276884153","cluster_addr":"172.21.15.28:6811/276884153","heartbeat_back_addr":"172.21.15.28:6815/276884153","heartbeat_front_addr":"172.21.15.28:6813/276884153","state":["exists","up"]},{"osd":5,"uuid":"a2bf6f7d-170e-4b63-bf7a-8562ad2b66a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2643308264},{"type":"v1","addr":"172.21.15.28:6817","nonce":2643308264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":2643308264},{"type":"v1","addr":"172.21.15.28:6819","nonce":2643308264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":2643308264},{"type":"v1","addr":"172.21.15.28:6823","nonce":2643308264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2643308264},{"type":"v1","addr":"172.21.15.28:6821","nonce":2643308264}]},"public_addr":"172.21.15.28:6817/2643308264","cluster_addr":"172.21.15.28:6819/2643308264","heartbeat_back_addr":"172.21.15.28:6823/2643308264","heartbeat_front_addr":"172.21.15.28:6821/2643308264","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-18T02:33:14.758941+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-18T02:33:41.399768+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-18T02:34:08.356814+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-18T02:34:31.488432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-18T02:34:56.105704+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-18T02:35:20.190437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/776743141":"2024-06-19T02:32:06.496885+0000","172.21.15.1:0/2875918960":"2024-06-19T02:32:06.496885+0000","172.21.15.1:6801/752464819":"2024-06-19T02:30:29.235259+0000","172.21.15.1:6801/4268462134":"2024-06-19T02:30:58.990073+0000","172.21.15.1:0/1466863637":"2024-06-19T02:30:29.235259+0000","172.21.15.1:0/1245204308":"2024-06-19T02:30:29.235259+0000","172.21.15.1:0/2844026642":"2024-06-19T02:30:58.990073+0000","172.21.15.1:0/138615605":"2024-06-19T02:30:58.990073+0000","172.21.15.1:0/2566240088":"2024-06-19T02:30:29.235259+0000","172.21.15.1:0/2788805455":"2024-06-19T02:30:58.990073+0000","172.21.15.1:6801/85550937":"2024-06-19T02:32:06.496885+0000","172.21.15.1:0/258317390":"2024-06-19T02:32:06.496885+0000","172.21.15.1:6800/752464819":"2024-06-19T02:30:29.235259+0000","172.21.15.1:6800/4268462134":"2024-06-19T02:30:58.990073+0000","172.21.15.1:6800/85550937":"2024-06-19T02:32:06.496885+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-18T02:35:43.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:43.330+0000 7fa1bc542640 1 -- 172.21.15.1:0/2187291769 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fa1a0065430 msgr2=0x7fa1a00678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:35:43.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:43.330+0000 7fa1bc542640 1 --2- 172.21.15.1:0/2187291769 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fa1a0065430 0x7fa1a00678e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa1b4110ae0 tx=0x7fa1a400b040 comp rx=0 tx=0).stop 2024-06-18T02:35:43.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:43.330+0000 7fa1bc542640 1 -- 172.21.15.1:0/2187291769 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1b41022a0 msgr2=0x7fa1b4115220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:35:43.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:43.330+0000 7fa1bc542640 1 --2- 172.21.15.1:0/2187291769 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1b41022a0 0x7fa1b4115220 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fa1b00043d0 tx=0x7fa1b0009da0 comp rx=0 tx=0).stop 2024-06-18T02:35:43.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:43.330+0000 7fa1bc542640 1 -- 172.21.15.1:0/2187291769 shutdown_connections 2024-06-18T02:35:43.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:43.330+0000 7fa1bc542640 1 --2- 172.21.15.1:0/2187291769 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fa1a0065430 0x7fa1a00678e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:43.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:43.330+0000 7fa1bc542640 1 --2- 172.21.15.1:0/2187291769 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1b4102c40 0x7fa1b4115760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:43.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:43.330+0000 7fa1bc542640 1 --2- 172.21.15.1:0/2187291769 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa1b41022a0 0x7fa1b4115220 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:43.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:43.330+0000 7fa1bc542640 1 -- 172.21.15.1:0/2187291769 >> 172.21.15.1:0/2187291769 conn(0x7fa1b40f7e50 msgr2=0x7fa1b4071130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:35:43.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:43.330+0000 7fa1bc542640 1 -- 172.21.15.1:0/2187291769 shutdown_connections 2024-06-18T02:35:43.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:43.330+0000 7fa1bc542640 1 -- 172.21.15.1:0/2187291769 wait complete. 2024-06-18T02:35:44.081 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:43 smithi001 ceph-mon[29099]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:35:44.081 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:43 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/2187291769' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-18T02:35:44.111 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-06-18T02:35:44.111 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph osd dump --format=json 2024-06-18T02:35:44.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:43 smithi028 ceph-mon[35823]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:35:44.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:43 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/2187291769' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-18T02:35:44.377 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:35:45.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:45.981+0000 7f13a4895640 1 -- 172.21.15.1:0/311485429 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13a01022a0 msgr2=0x7f13a0102670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:35:45.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:45.981+0000 7f13a4895640 1 --2- 172.21.15.1:0/311485429 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13a01022a0 0x7f13a0102670 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f13900048d0 tx=0x7f1390009da0 comp rx=0 tx=0).stop 2024-06-18T02:35:45.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:45.981+0000 7f13a4895640 1 -- 172.21.15.1:0/311485429 shutdown_connections 2024-06-18T02:35:45.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:45.981+0000 7f13a4895640 1 --2- 172.21.15.1:0/311485429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13a0102c40 0x7f13a00fc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:45.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:45.981+0000 7f13a4895640 1 --2- 172.21.15.1:0/311485429 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13a01022a0 0x7f13a0102670 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:45.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:45.981+0000 7f13a4895640 1 -- 172.21.15.1:0/311485429 >> 172.21.15.1:0/311485429 conn(0x7f13a00f7e50 msgr2=0x7f13a00fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:35:45.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:45.981+0000 7f13a4895640 1 -- 172.21.15.1:0/311485429 shutdown_connections 2024-06-18T02:35:45.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:45.981+0000 7f13a4895640 1 -- 172.21.15.1:0/311485429 wait complete. 2024-06-18T02:35:45.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:45.981+0000 7f13a4895640 1 Processor -- start 2024-06-18T02:35:45.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:45.982+0000 7f13a4895640 1 -- start start 2024-06-18T02:35:45.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:45.982+0000 7f13a4895640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13a01022a0 0x7f13a0112fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:35:45.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:45.982+0000 7f13a4895640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13a0102c40 0x7f13a01134f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:35:45.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:45.982+0000 7f13a4895640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13a0113af0 con 0x7f13a01022a0 2024-06-18T02:35:45.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:45.982+0000 7f13a4895640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13a010cf30 con 0x7f13a0102c40 2024-06-18T02:35:45.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:45.982+0000 7f139e59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13a01022a0 0x7f13a0112fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:35:45.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:45.982+0000 7f139e59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13a01022a0 0x7f13a0112fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47808/0 (socket says 172.21.15.1:47808) 2024-06-18T02:35:45.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:45.982+0000 7f139e59c640 1 -- 172.21.15.1:0/392984837 learned_addr learned my addr 172.21.15.1:0/392984837 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:35:45.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:45.982+0000 7f139dd9b640 1 --2- 172.21.15.1:0/392984837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13a0102c40 0x7f13a01134f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:35:45.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:45.982+0000 7f139e59c640 1 -- 172.21.15.1:0/392984837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13a0102c40 msgr2=0x7f13a01134f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:35:45.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:45.982+0000 7f139e59c640 1 --2- 172.21.15.1:0/392984837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13a0102c40 0x7f13a01134f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:45.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:45.982+0000 7f139e59c640 1 -- 172.21.15.1:0/392984837 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1390005040 con 0x7f13a01022a0 2024-06-18T02:35:45.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:45.982+0000 7f139e59c640 1 --2- 172.21.15.1:0/392984837 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13a01022a0 0x7f13a0112fb0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1390003080 tx=0x7f1390009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:35:45.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:45.983+0000 7f138effd640 1 -- 172.21.15.1:0/392984837 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1390038070 con 0x7f13a01022a0 2024-06-18T02:35:45.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:45.983+0000 7f138effd640 1 -- 172.21.15.1:0/392984837 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1390008ee0 con 0x7f13a01022a0 2024-06-18T02:35:45.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:45.983+0000 7f13a4895640 1 -- 172.21.15.1:0/392984837 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13a010d1b0 con 0x7f13a01022a0 2024-06-18T02:35:45.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:45.983+0000 7f138effd640 1 -- 172.21.15.1:0/392984837 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f139003d870 con 0x7f13a01022a0 2024-06-18T02:35:45.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:45.983+0000 7f13a4895640 1 -- 172.21.15.1:0/392984837 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13a010d6f0 con 0x7f13a01022a0 2024-06-18T02:35:45.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:45.984+0000 7f13a4895640 1 -- 172.21.15.1:0/392984837 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13a00fda10 con 0x7f13a01022a0 2024-06-18T02:35:45.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:45.984+0000 7f138effd640 1 -- 172.21.15.1:0/392984837 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7f1390030020 con 0x7f13a01022a0 2024-06-18T02:35:45.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:45.984+0000 7f138effd640 1 --2- 172.21.15.1:0/392984837 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f1388065430 0x7f13880678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:35:45.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:45.985+0000 7f138effd640 1 -- 172.21.15.1:0/392984837 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f13900a2070 con 0x7f13a01022a0 2024-06-18T02:35:45.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:45.985+0000 7f139dd9b640 1 --2- 172.21.15.1:0/392984837 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f1388065430 0x7f13880678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:35:45.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:45.985+0000 7f139dd9b640 1 --2- 172.21.15.1:0/392984837 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f1388065430 0x7f13880678e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f13a010e870 tx=0x7f1394009040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:35:45.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:45.989+0000 7f138effd640 1 -- 172.21.15.1:0/392984837 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f1390070640 con 0x7f13a01022a0 2024-06-18T02:35:46.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:45 smithi001 ceph-mon[29099]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:35:46.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:46.164+0000 7f13a4895640 1 -- 172.21.15.1:0/392984837 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f13a004b480 con 0x7f13a01022a0 2024-06-18T02:35:46.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:46.165+0000 7f138effd640 1 -- 172.21.15.1:0/392984837 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+10965 (secure 0 0 0) 0x7f1390073c60 con 0x7f13a01022a0 2024-06-18T02:35:46.166 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:35:46.166 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":40,"fsid":"7eefb63e-2d1a-11ef-bc9d-c7b262605968","created":"2024-06-18T02:29:58.467010+0000","modified":"2024-06-18T02:35:22.963059+0000","last_up_change":"2024-06-18T02:35:21.961141+0000","last_in_change":"2024-06-18T02:35:03.501632+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-18T02:34:11.007633+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"2fc85253-858e-496d-a262-061526504dc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":797495043},{"type":"v1","addr":"172.21.15.1:6803","nonce":797495043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":797495043},{"type":"v1","addr":"172.21.15.1:6805","nonce":797495043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":797495043},{"type":"v1","addr":"172.21.15.1:6809","nonce":797495043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":797495043},{"type":"v1","addr":"172.21.15.1:6807","nonce":797495043}]},"public_addr":"172.21.15.1:6803/797495043","cluster_addr":"172.21.15.1:6805/797495043","heartbeat_back_addr":"172.21.15.1:6809/797495043","heartbeat_front_addr":"172.21.15.1:6807/797495043","state":["exists","up"]},{"osd":1,"uuid":"f9bb7834-4fa4-40b1-9b63-bd1a69a4e082","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":2182416687},{"type":"v1","addr":"172.21.15.1:6811","nonce":2182416687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6812","nonce":2182416687},{"type":"v1","addr":"172.21.15.1:6813","nonce":2182416687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6816","nonce":2182416687},{"type":"v1","addr":"172.21.15.1:6817","nonce":2182416687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":2182416687},{"type":"v1","addr":"172.21.15.1:6815","nonce":2182416687}]},"public_addr":"172.21.15.1:6811/2182416687","cluster_addr":"172.21.15.1:6813/2182416687","heartbeat_back_addr":"172.21.15.1:6817/2182416687","heartbeat_front_addr":"172.21.15.1:6815/2182416687","state":["exists","up"]},{"osd":2,"uuid":"89223bec-bba5-4744-960e-5396f04ddba1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":3682942580},{"type":"v1","addr":"172.21.15.1:6819","nonce":3682942580}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6820","nonce":3682942580},{"type":"v1","addr":"172.21.15.1:6821","nonce":3682942580}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":3682942580},{"type":"v1","addr":"172.21.15.1:6825","nonce":3682942580}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":3682942580},{"type":"v1","addr":"172.21.15.1:6823","nonce":3682942580}]},"public_addr":"172.21.15.1:6819/3682942580","cluster_addr":"172.21.15.1:6821/3682942580","heartbeat_back_addr":"172.21.15.1:6825/3682942580","heartbeat_front_addr":"172.21.15.1:6823/3682942580","state":["exists","up"]},{"osd":3,"uuid":"33dd638d-10d2-432a-92ac-74b68b4a8311","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":664150046},{"type":"v1","addr":"172.21.15.28:6801","nonce":664150046}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":664150046},{"type":"v1","addr":"172.21.15.28:6803","nonce":664150046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":664150046},{"type":"v1","addr":"172.21.15.28:6807","nonce":664150046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":664150046},{"type":"v1","addr":"172.21.15.28:6805","nonce":664150046}]},"public_addr":"172.21.15.28:6801/664150046","cluster_addr":"172.21.15.28:6803/664150046","heartbeat_back_addr":"172.21.15.28:6807/664150046","heartbeat_front_addr":"172.21.15.28:6805/664150046","state":["exists","up"]},{"osd":4,"uuid":"2f4a6526-2f3e-417f-b02c-cf242e7eac07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":276884153},{"type":"v1","addr":"172.21.15.28:6809","nonce":276884153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":276884153},{"type":"v1","addr":"172.21.15.28:6811","nonce":276884153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":276884153},{"type":"v1","addr":"172.21.15.28:6815","nonce":276884153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":276884153},{"type":"v1","addr":"172.21.15.28:6813","nonce":276884153}]},"public_addr":"172.21.15.28:6809/276884153","cluster_addr":"172.21.15.28:6811/276884153","heartbeat_back_addr":"172.21.15.28:6815/276884153","heartbeat_front_addr":"172.21.15.28:6813/276884153","state":["exists","up"]},{"osd":5,"uuid":"a2bf6f7d-170e-4b63-bf7a-8562ad2b66a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2643308264},{"type":"v1","addr":"172.21.15.28:6817","nonce":2643308264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":2643308264},{"type":"v1","addr":"172.21.15.28:6819","nonce":2643308264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":2643308264},{"type":"v1","addr":"172.21.15.28:6823","nonce":2643308264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":2643308264},{"type":"v1","addr":"172.21.15.28:6821","nonce":2643308264}]},"public_addr":"172.21.15.28:6817/2643308264","cluster_addr":"172.21.15.28:6819/2643308264","heartbeat_back_addr":"172.21.15.28:6823/2643308264","heartbeat_front_addr":"172.21.15.28:6821/2643308264","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-18T02:33:14.758941+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-18T02:33:41.399768+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-18T02:34:08.356814+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-18T02:34:31.488432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-18T02:34:56.105704+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-18T02:35:20.190437+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.1:0/776743141":"2024-06-19T02:32:06.496885+0000","172.21.15.1:0/2875918960":"2024-06-19T02:32:06.496885+0000","172.21.15.1:6801/752464819":"2024-06-19T02:30:29.235259+0000","172.21.15.1:6801/4268462134":"2024-06-19T02:30:58.990073+0000","172.21.15.1:0/1466863637":"2024-06-19T02:30:29.235259+0000","172.21.15.1:0/1245204308":"2024-06-19T02:30:29.235259+0000","172.21.15.1:0/2844026642":"2024-06-19T02:30:58.990073+0000","172.21.15.1:0/138615605":"2024-06-19T02:30:58.990073+0000","172.21.15.1:0/2566240088":"2024-06-19T02:30:29.235259+0000","172.21.15.1:0/2788805455":"2024-06-19T02:30:58.990073+0000","172.21.15.1:6801/85550937":"2024-06-19T02:32:06.496885+0000","172.21.15.1:0/258317390":"2024-06-19T02:32:06.496885+0000","172.21.15.1:6800/752464819":"2024-06-19T02:30:29.235259+0000","172.21.15.1:6800/4268462134":"2024-06-19T02:30:58.990073+0000","172.21.15.1:6800/85550937":"2024-06-19T02:32:06.496885+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-18T02:35:46.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:46.167+0000 7f13a4895640 1 -- 172.21.15.1:0/392984837 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f1388065430 msgr2=0x7f13880678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:35:46.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:46.167+0000 7f13a4895640 1 --2- 172.21.15.1:0/392984837 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f1388065430 0x7f13880678e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f13a010e870 tx=0x7f1394009040 comp rx=0 tx=0).stop 2024-06-18T02:35:46.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:46.167+0000 7f13a4895640 1 -- 172.21.15.1:0/392984837 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13a01022a0 msgr2=0x7f13a0112fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:35:46.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:46.167+0000 7f13a4895640 1 --2- 172.21.15.1:0/392984837 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13a01022a0 0x7f13a0112fb0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1390003080 tx=0x7f1390009da0 comp rx=0 tx=0).stop 2024-06-18T02:35:46.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:46.168+0000 7f13a4895640 1 -- 172.21.15.1:0/392984837 shutdown_connections 2024-06-18T02:35:46.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:46.168+0000 7f13a4895640 1 --2- 172.21.15.1:0/392984837 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f1388065430 0x7f13880678e0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:46.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:46.168+0000 7f13a4895640 1 --2- 172.21.15.1:0/392984837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13a0102c40 0x7f13a01134f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:46.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:46.168+0000 7f13a4895640 1 --2- 172.21.15.1:0/392984837 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f13a01022a0 0x7f13a0112fb0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:46.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:46.168+0000 7f13a4895640 1 -- 172.21.15.1:0/392984837 >> 172.21.15.1:0/392984837 conn(0x7f13a00f7e50 msgr2=0x7f13a00f8db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:35:46.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:46.168+0000 7f13a4895640 1 -- 172.21.15.1:0/392984837 shutdown_connections 2024-06-18T02:35:46.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:46.168+0000 7f13a4895640 1 -- 172.21.15.1:0/392984837 wait complete. 2024-06-18T02:35:46.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:45 smithi028 ceph-mon[35823]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:35:46.839 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-06-18T02:35:46.840 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-06-18T02:35:46.840 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-06-18T02:35:46.841 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-06-18T02:35:46.841 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-06-18T02:35:46.842 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-06-18T02:35:47.127 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:46 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/392984837' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-18T02:35:47.164 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:35:47.165 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:35:47.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:46 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/392984837' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-18T02:35:47.875 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:47 smithi001 ceph-mon[29099]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:35:48.144 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:47 smithi028 ceph-mon[35823]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:35:49.648 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:35:49.649 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:35:49.651 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:35:50.010 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:49 smithi001 ceph-mon[29099]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:35:50.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:49 smithi028 ceph-mon[35823]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:35:51.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:51.192+0000 7f7fe51dc640 1 -- 172.21.15.1:0/2573406915 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fe00fbe80 msgr2=0x7f7fe00fcd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:35:51.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:51.192+0000 7f7fe51dc640 1 --2- 172.21.15.1:0/2573406915 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fe00fbe80 0x7f7fe00fcd80 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f7fc80048d0 tx=0x7f7fc8009da0 comp rx=0 tx=0).stop 2024-06-18T02:35:51.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:51.192+0000 7f7fe51dc640 1 -- 172.21.15.1:0/2573406915 shutdown_connections 2024-06-18T02:35:51.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:51.192+0000 7f7fe51dc640 1 --2- 172.21.15.1:0/2573406915 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fe00fbe80 0x7f7fe00fcd80 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:51.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:51.192+0000 7f7fe51dc640 1 --2- 172.21.15.1:0/2573406915 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fe00fb570 0x7f7fe00fb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:51.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:51.192+0000 7f7fe51dc640 1 -- 172.21.15.1:0/2573406915 >> 172.21.15.1:0/2573406915 conn(0x7f7fe0072110 msgr2=0x7f7fe0072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:35:51.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:51.192+0000 7f7fe51dc640 1 -- 172.21.15.1:0/2573406915 shutdown_connections 2024-06-18T02:35:51.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:51.192+0000 7f7fe51dc640 1 -- 172.21.15.1:0/2573406915 wait complete. 2024-06-18T02:35:51.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:51.193+0000 7f7fe51dc640 1 Processor -- start 2024-06-18T02:35:51.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:51.193+0000 7f7fe51dc640 1 -- start start 2024-06-18T02:35:51.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:51.193+0000 7f7fe51dc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fe00fb570 0x7f7fe0112fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:35:51.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:51.193+0000 7f7fe51dc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fe00fbe80 0x7f7fe01134f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:35:51.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:51.193+0000 7f7fe51dc640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe0113af0 con 0x7f7fe00fbe80 2024-06-18T02:35:51.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:51.193+0000 7f7fe51dc640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe010cf30 con 0x7f7fe00fb570 2024-06-18T02:35:51.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:51.193+0000 7f7fd7fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fe00fbe80 0x7f7fe01134f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:35:51.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:51.193+0000 7f7fd7fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fe00fbe80 0x7f7fe01134f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47830/0 (socket says 172.21.15.1:47830) 2024-06-18T02:35:51.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:51.193+0000 7f7fd7fff640 1 -- 172.21.15.1:0/3942442706 learned_addr learned my addr 172.21.15.1:0/3942442706 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:35:51.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:51.194+0000 7f7fded9d640 1 --2- 172.21.15.1:0/3942442706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fe00fb570 0x7f7fe0112fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:35:51.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:51.194+0000 7f7fd7fff640 1 -- 172.21.15.1:0/3942442706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fe00fb570 msgr2=0x7f7fe0112fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:35:51.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:51.194+0000 7f7fd7fff640 1 --2- 172.21.15.1:0/3942442706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fe00fb570 0x7f7fe0112fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:51.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:51.194+0000 7f7fd7fff640 1 -- 172.21.15.1:0/3942442706 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fc8005040 con 0x7f7fe00fbe80 2024-06-18T02:35:51.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:51.194+0000 7f7fd7fff640 1 --2- 172.21.15.1:0/3942442706 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fe00fbe80 0x7f7fe01134f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f7fc8003080 tx=0x7f7fc8009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:35:51.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:51.194+0000 7f7fd77fe640 1 -- 172.21.15.1:0/3942442706 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7fc8038070 con 0x7f7fe00fbe80 2024-06-18T02:35:51.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:51.194+0000 7f7fd77fe640 1 -- 172.21.15.1:0/3942442706 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7fc8008ee0 con 0x7f7fe00fbe80 2024-06-18T02:35:51.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:51.194+0000 7f7fe51dc640 1 -- 172.21.15.1:0/3942442706 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fe010d1b0 con 0x7f7fe00fbe80 2024-06-18T02:35:51.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:51.194+0000 7f7fd77fe640 1 -- 172.21.15.1:0/3942442706 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7fc803d870 con 0x7f7fe00fbe80 2024-06-18T02:35:51.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:51.194+0000 7f7fe51dc640 1 -- 172.21.15.1:0/3942442706 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fe010d6f0 con 0x7f7fe00fbe80 2024-06-18T02:35:51.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:51.195+0000 7f7fe51dc640 1 -- 172.21.15.1:0/3942442706 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7fe010dc90 con 0x7f7fe00fbe80 2024-06-18T02:35:51.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:51.196+0000 7f7fd77fe640 1 -- 172.21.15.1:0/3942442706 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7f7fc8030020 con 0x7f7fe00fbe80 2024-06-18T02:35:51.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:51.196+0000 7f7fd77fe640 1 --2- 172.21.15.1:0/3942442706 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f7fcc065430 0x7f7fcc0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:35:51.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:51.196+0000 7f7fd77fe640 1 -- 172.21.15.1:0/3942442706 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f7fc80a2230 con 0x7f7fe00fbe80 2024-06-18T02:35:51.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:51.196+0000 7f7fded9d640 1 --2- 172.21.15.1:0/3942442706 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f7fcc065430 0x7f7fcc0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:35:51.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:51.196+0000 7f7fd77fe640 1 --2- 172.21.15.1:0/3942442706 >> [v2:172.21.15.1:6802/797495043,v1:172.21.15.1:6803/797495043] conn(0x7f7fcc06b1b0 0x7f7fcc06d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:35:51.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:51.196+0000 7f7fd77fe640 1 -- 172.21.15.1:0/3942442706 --> [v2:172.21.15.1:6802/797495043,v1:172.21.15.1:6803/797495043] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7fcc06dc70 con 0x7f7fcc06b1b0 2024-06-18T02:35:51.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:51.196+0000 7f7fd77fe640 1 -- 172.21.15.1:0/3942442706 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7fc80a4820 con 0x7f7fe00fbe80 2024-06-18T02:35:51.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:51.196+0000 7f7fdf59e640 1 --2- 172.21.15.1:0/3942442706 >> [v2:172.21.15.1:6802/797495043,v1:172.21.15.1:6803/797495043] conn(0x7f7fcc06b1b0 0x7f7fcc06d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:35:51.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:51.197+0000 7f7fded9d640 1 --2- 172.21.15.1:0/3942442706 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f7fcc065430 0x7f7fcc0678e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7fd0004b20 tx=0x7f7fd0009040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:35:51.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:51.197+0000 7f7fdf59e640 1 --2- 172.21.15.1:0/3942442706 >> [v2:172.21.15.1:6802/797495043,v1:172.21.15.1:6803/797495043] conn(0x7f7fcc06b1b0 0x7f7fcc06d5c0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:35:51.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:51.197+0000 7f7fd77fe640 1 -- 172.21.15.1:0/3942442706 <== osd.0 v2:172.21.15.1:6802/797495043 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f7fcc06dc70 con 0x7f7fcc06b1b0 2024-06-18T02:35:51.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:51.211+0000 7f7fe51dc640 1 -- 172.21.15.1:0/3942442706 --> [v2:172.21.15.1:6802/797495043,v1:172.21.15.1:6803/797495043] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7fe005dbb0 con 0x7f7fcc06b1b0 2024-06-18T02:35:51.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:51.211+0000 7f7fd77fe640 1 -- 172.21.15.1:0/3942442706 <== osd.0 v2:172.21.15.1:6802/797495043 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f7fe005dbb0 con 0x7f7fcc06b1b0 2024-06-18T02:35:51.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:51.211+0000 7f7fe51dc640 1 -- 172.21.15.1:0/3942442706 >> [v2:172.21.15.1:6802/797495043,v1:172.21.15.1:6803/797495043] conn(0x7f7fcc06b1b0 msgr2=0x7f7fcc06d5c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:35:51.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:51.211+0000 7f7fe51dc640 1 --2- 172.21.15.1:0/3942442706 >> [v2:172.21.15.1:6802/797495043,v1:172.21.15.1:6803/797495043] conn(0x7f7fcc06b1b0 0x7f7fcc06d5c0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:51.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:51.211+0000 7f7fe51dc640 1 -- 172.21.15.1:0/3942442706 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f7fcc065430 msgr2=0x7f7fcc0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:35:51.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:51.211+0000 7f7fe51dc640 1 --2- 172.21.15.1:0/3942442706 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f7fcc065430 0x7f7fcc0678e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7fd0004b20 tx=0x7f7fd0009040 comp rx=0 tx=0).stop 2024-06-18T02:35:51.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:51.211+0000 7f7fe51dc640 1 -- 172.21.15.1:0/3942442706 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fe00fbe80 msgr2=0x7f7fe01134f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:35:51.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:51.211+0000 7f7fe51dc640 1 --2- 172.21.15.1:0/3942442706 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fe00fbe80 0x7f7fe01134f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f7fc8003080 tx=0x7f7fc8009da0 comp rx=0 tx=0).stop 2024-06-18T02:35:51.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:51.211+0000 7f7fe51dc640 1 -- 172.21.15.1:0/3942442706 shutdown_connections 2024-06-18T02:35:51.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:51.211+0000 7f7fe51dc640 1 --2- 172.21.15.1:0/3942442706 >> [v2:172.21.15.1:6802/797495043,v1:172.21.15.1:6803/797495043] conn(0x7f7fcc06b1b0 0x7f7fcc06d5c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:51.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:51.211+0000 7f7fe51dc640 1 --2- 172.21.15.1:0/3942442706 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f7fcc065430 0x7f7fcc0678e0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:51.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:51.211+0000 7f7fe51dc640 1 --2- 172.21.15.1:0/3942442706 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fe00fbe80 0x7f7fe01134f0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:51.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:51.211+0000 7f7fe51dc640 1 --2- 172.21.15.1:0/3942442706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fe00fb570 0x7f7fe0112fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:51.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:51.211+0000 7f7fe51dc640 1 -- 172.21.15.1:0/3942442706 >> 172.21.15.1:0/3942442706 conn(0x7f7fe0072110 msgr2=0x7f7fe01000a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:35:51.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:51.212+0000 7f7fe51dc640 1 -- 172.21.15.1:0/3942442706 shutdown_connections 2024-06-18T02:35:51.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:51.212+0000 7f7fe51dc640 1 -- 172.21.15.1:0/3942442706 wait complete. 2024-06-18T02:35:52.127 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:51 smithi001 ceph-mon[29099]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:35:52.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:51 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:35:52.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:51 smithi028 ceph-mon[35823]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:35:52.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:51 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:35:53.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.868+0000 7f947da3c640 1 -- 172.21.15.1:0/540396711 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9478102c90 msgr2=0x7f94780fa9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:35:53.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.868+0000 7f947da3c640 1 --2- 172.21.15.1:0/540396711 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9478102c90 0x7f94780fa9e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f94600048d0 tx=0x7f9460009da0 comp rx=0 tx=0).stop 2024-06-18T02:35:53.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.868+0000 7f947da3c640 1 -- 172.21.15.1:0/540396711 shutdown_connections 2024-06-18T02:35:53.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.868+0000 7f947da3c640 1 --2- 172.21.15.1:0/540396711 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9478102c90 0x7f94780fa9e0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:53.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.868+0000 7f947da3c640 1 --2- 172.21.15.1:0/540396711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94781022f0 0x7f94781026c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:53.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.868+0000 7f947da3c640 1 -- 172.21.15.1:0/540396711 >> 172.21.15.1:0/540396711 conn(0x7f9478070a40 msgr2=0x7f9478070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:35:53.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.868+0000 7f947da3c640 1 -- 172.21.15.1:0/540396711 shutdown_connections 2024-06-18T02:35:53.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.868+0000 7f947da3c640 1 -- 172.21.15.1:0/540396711 wait complete. 2024-06-18T02:35:53.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.868+0000 7f947da3c640 1 Processor -- start 2024-06-18T02:35:53.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.868+0000 7f947da3c640 1 -- start start 2024-06-18T02:35:53.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.869+0000 7f947da3c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94781022f0 0x7f94781152f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:35:53.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.869+0000 7f947da3c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9478102c90 0x7f9478115830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:35:53.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.869+0000 7f947da3c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9478115e30 con 0x7f94781022f0 2024-06-18T02:35:53.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.869+0000 7f947da3c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f947810f270 con 0x7f9478102c90 2024-06-18T02:35:53.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.869+0000 7f9476ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94781022f0 0x7f94781152f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:35:53.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.869+0000 7f9476ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94781022f0 0x7f94781152f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40104/0 (socket says 172.21.15.1:40104) 2024-06-18T02:35:53.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.869+0000 7f9476ffd640 1 -- 172.21.15.1:0/895673795 learned_addr learned my addr 172.21.15.1:0/895673795 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:35:53.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.869+0000 7f946ffff640 1 --2- 172.21.15.1:0/895673795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9478102c90 0x7f9478115830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:35:53.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.869+0000 7f9476ffd640 1 -- 172.21.15.1:0/895673795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9478102c90 msgr2=0x7f9478115830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:35:53.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.869+0000 7f9476ffd640 1 --2- 172.21.15.1:0/895673795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9478102c90 0x7f9478115830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:53.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.869+0000 7f9476ffd640 1 -- 172.21.15.1:0/895673795 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9460005040 con 0x7f94781022f0 2024-06-18T02:35:53.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.869+0000 7f9476ffd640 1 --2- 172.21.15.1:0/895673795 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94781022f0 0x7f94781152f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f9468002840 tx=0x7f9468004570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:35:53.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.869+0000 7f946f7fe640 1 -- 172.21.15.1:0/895673795 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9468007c30 con 0x7f94781022f0 2024-06-18T02:35:53.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.869+0000 7f946f7fe640 1 -- 172.21.15.1:0/895673795 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f946800f040 con 0x7f94781022f0 2024-06-18T02:35:53.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.869+0000 7f946f7fe640 1 -- 172.21.15.1:0/895673795 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9468009570 con 0x7f94781022f0 2024-06-18T02:35:53.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.870+0000 7f947da3c640 1 -- 172.21.15.1:0/895673795 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f947810f550 con 0x7f94781022f0 2024-06-18T02:35:53.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.870+0000 7f947da3c640 1 -- 172.21.15.1:0/895673795 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f947810fa40 con 0x7f94781022f0 2024-06-18T02:35:53.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.870+0000 7f947da3c640 1 -- 172.21.15.1:0/895673795 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f947810ff40 con 0x7f94781022f0 2024-06-18T02:35:53.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.871+0000 7f946f7fe640 1 -- 172.21.15.1:0/895673795 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7f946800dc80 con 0x7f94781022f0 2024-06-18T02:35:53.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.871+0000 7f946f7fe640 1 --2- 172.21.15.1:0/895673795 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f9464065430 0x7f94640678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:35:53.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.871+0000 7f946f7fe640 1 -- 172.21.15.1:0/895673795 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f9468082380 con 0x7f94781022f0 2024-06-18T02:35:53.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.872+0000 7f946ffff640 1 --2- 172.21.15.1:0/895673795 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f9464065430 0x7f94640678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:35:53.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.872+0000 7f946f7fe640 1 --2- 172.21.15.1:0/895673795 >> [v2:172.21.15.1:6818/3682942580,v1:172.21.15.1:6819/3682942580] conn(0x7f946406b1b0 0x7f946406d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:35:53.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.872+0000 7f946f7fe640 1 -- 172.21.15.1:0/895673795 --> [v2:172.21.15.1:6818/3682942580,v1:172.21.15.1:6819/3682942580] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f946406dc70 con 0x7f946406b1b0 2024-06-18T02:35:53.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.872+0000 7f946f7fe640 1 -- 172.21.15.1:0/895673795 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f94680849d0 con 0x7f94781022f0 2024-06-18T02:35:53.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.872+0000 7f94777fe640 1 --2- 172.21.15.1:0/895673795 >> [v2:172.21.15.1:6818/3682942580,v1:172.21.15.1:6819/3682942580] conn(0x7f946406b1b0 0x7f946406d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:35:53.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.872+0000 7f946ffff640 1 --2- 172.21.15.1:0/895673795 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f9464065430 0x7f94640678e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f94600048a0 tx=0x7f9460035040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:35:53.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.872+0000 7f94777fe640 1 --2- 172.21.15.1:0/895673795 >> [v2:172.21.15.1:6818/3682942580,v1:172.21.15.1:6819/3682942580] conn(0x7f946406b1b0 0x7f946406d5c0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:35:53.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.873+0000 7f946f7fe640 1 -- 172.21.15.1:0/895673795 <== osd.2 v2:172.21.15.1:6818/3682942580 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f946406dc70 con 0x7f946406b1b0 2024-06-18T02:35:53.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.887+0000 7f947da3c640 1 -- 172.21.15.1:0/895673795 --> [v2:172.21.15.1:6818/3682942580,v1:172.21.15.1:6819/3682942580] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f94781100b0 con 0x7f946406b1b0 2024-06-18T02:35:53.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.887+0000 7f946f7fe640 1 -- 172.21.15.1:0/895673795 <== osd.2 v2:172.21.15.1:6818/3682942580 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f94781100b0 con 0x7f946406b1b0 2024-06-18T02:35:53.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.888+0000 7f947da3c640 1 -- 172.21.15.1:0/895673795 >> [v2:172.21.15.1:6818/3682942580,v1:172.21.15.1:6819/3682942580] conn(0x7f946406b1b0 msgr2=0x7f946406d5c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:35:53.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.888+0000 7f947da3c640 1 --2- 172.21.15.1:0/895673795 >> [v2:172.21.15.1:6818/3682942580,v1:172.21.15.1:6819/3682942580] conn(0x7f946406b1b0 0x7f946406d5c0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:53.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.888+0000 7f947da3c640 1 -- 172.21.15.1:0/895673795 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f9464065430 msgr2=0x7f94640678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:35:53.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.888+0000 7f947da3c640 1 --2- 172.21.15.1:0/895673795 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f9464065430 0x7f94640678e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f94600048a0 tx=0x7f9460035040 comp rx=0 tx=0).stop 2024-06-18T02:35:53.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.888+0000 7f947da3c640 1 -- 172.21.15.1:0/895673795 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94781022f0 msgr2=0x7f94781152f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:35:53.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.888+0000 7f947da3c640 1 --2- 172.21.15.1:0/895673795 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94781022f0 0x7f94781152f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f9468002840 tx=0x7f9468004570 comp rx=0 tx=0).stop 2024-06-18T02:35:53.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.888+0000 7f947da3c640 1 -- 172.21.15.1:0/895673795 shutdown_connections 2024-06-18T02:35:53.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.888+0000 7f947da3c640 1 --2- 172.21.15.1:0/895673795 >> [v2:172.21.15.1:6818/3682942580,v1:172.21.15.1:6819/3682942580] conn(0x7f946406b1b0 0x7f946406d5c0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:53.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.888+0000 7f947da3c640 1 --2- 172.21.15.1:0/895673795 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f9464065430 0x7f94640678e0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:53.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.888+0000 7f947da3c640 1 --2- 172.21.15.1:0/895673795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9478102c90 0x7f9478115830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:53.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.888+0000 7f947da3c640 1 --2- 172.21.15.1:0/895673795 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94781022f0 0x7f94781152f0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:53.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.888+0000 7f947da3c640 1 -- 172.21.15.1:0/895673795 >> 172.21.15.1:0/895673795 conn(0x7f9478070a40 msgr2=0x7f94781030a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:35:53.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.888+0000 7f947da3c640 1 -- 172.21.15.1:0/895673795 shutdown_connections 2024-06-18T02:35:53.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.888+0000 7f947da3c640 1 -- 172.21.15.1:0/895673795 wait complete. 2024-06-18T02:35:54.107 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:53 smithi001 ceph-mon[29099]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:35:54.108 INFO:teuthology.orchestra.run.smithi001.stdout:38654705697 2024-06-18T02:35:54.109 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-06-18T02:35:54.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.451+0000 7fa4778c4640 1 -- 172.21.15.1:0/2074983643 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4700652f0 msgr2=0x7fa470100f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:35:54.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.451+0000 7fa4778c4640 1 --2- 172.21.15.1:0/2074983643 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4700652f0 0x7fa470100f30 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa4600048d0 tx=0x7fa460009da0 comp rx=0 tx=0).stop 2024-06-18T02:35:54.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.451+0000 7fa4778c4640 1 -- 172.21.15.1:0/2074983643 shutdown_connections 2024-06-18T02:35:54.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.451+0000 7fa4778c4640 1 --2- 172.21.15.1:0/2074983643 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4700652f0 0x7fa470100f30 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:54.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.451+0000 7fa4778c4640 1 --2- 172.21.15.1:0/2074983643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa470064f20 0x7fa4701009f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:54.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.451+0000 7fa4778c4640 1 -- 172.21.15.1:0/2074983643 >> 172.21.15.1:0/2074983643 conn(0x7fa470070a40 msgr2=0x7fa470070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:35:54.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.452+0000 7fa4778c4640 1 -- 172.21.15.1:0/2074983643 shutdown_connections 2024-06-18T02:35:54.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.452+0000 7fa4778c4640 1 -- 172.21.15.1:0/2074983643 wait complete. 2024-06-18T02:35:54.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.452+0000 7fa4778c4640 1 Processor -- start 2024-06-18T02:35:54.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.452+0000 7fa4778c4640 1 -- start start 2024-06-18T02:35:54.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.452+0000 7fa4778c4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa470064f20 0x7fa470113010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:35:54.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.452+0000 7fa4778c4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4700652f0 0x7fa470113550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:35:54.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.452+0000 7fa4778c4640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa470113be0 con 0x7fa470064f20 2024-06-18T02:35:54.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.452+0000 7fa4778c4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa47010cfd0 con 0x7fa4700652f0 2024-06-18T02:35:54.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.452+0000 7fa475660640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa470064f20 0x7fa470113010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:35:54.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.453+0000 7fa475660640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa470064f20 0x7fa470113010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47848/0 (socket says 172.21.15.1:47848) 2024-06-18T02:35:54.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.453+0000 7fa475660640 1 -- 172.21.15.1:0/1255863199 learned_addr learned my addr 172.21.15.1:0/1255863199 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:35:54.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.453+0000 7fa474e5f640 1 --2- 172.21.15.1:0/1255863199 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4700652f0 0x7fa470113550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:35:54.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.453+0000 7fa475660640 1 -- 172.21.15.1:0/1255863199 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4700652f0 msgr2=0x7fa470113550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:35:54.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.453+0000 7fa475660640 1 --2- 172.21.15.1:0/1255863199 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4700652f0 0x7fa470113550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:54.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.453+0000 7fa475660640 1 -- 172.21.15.1:0/1255863199 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa460005040 con 0x7fa470064f20 2024-06-18T02:35:54.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.453+0000 7fa475660640 1 --2- 172.21.15.1:0/1255863199 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa470064f20 0x7fa470113010 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa46c00b790 tx=0x7fa46c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:35:54.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.453+0000 7fa465ffb640 1 -- 172.21.15.1:0/1255863199 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa46c00c700 con 0x7fa470064f20 2024-06-18T02:35:54.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.453+0000 7fa4778c4640 1 -- 172.21.15.1:0/1255863199 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa47010d2b0 con 0x7fa470064f20 2024-06-18T02:35:54.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.453+0000 7fa465ffb640 1 -- 172.21.15.1:0/1255863199 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa46c00cd30 con 0x7fa470064f20 2024-06-18T02:35:54.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.453+0000 7fa465ffb640 1 -- 172.21.15.1:0/1255863199 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa46c013370 con 0x7fa470064f20 2024-06-18T02:35:54.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.453+0000 7fa4778c4640 1 -- 172.21.15.1:0/1255863199 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa47010d7d0 con 0x7fa470064f20 2024-06-18T02:35:54.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.454+0000 7fa4778c4640 1 -- 172.21.15.1:0/1255863199 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa47010dd10 con 0x7fa470064f20 2024-06-18T02:35:54.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.455+0000 7fa465ffb640 1 -- 172.21.15.1:0/1255863199 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7fa46c012070 con 0x7fa470064f20 2024-06-18T02:35:54.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.455+0000 7fa465ffb640 1 --2- 172.21.15.1:0/1255863199 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fa45c065430 0x7fa45c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:35:54.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.455+0000 7fa465ffb640 1 -- 172.21.15.1:0/1255863199 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fa46c0820c0 con 0x7fa470064f20 2024-06-18T02:35:54.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.455+0000 7fa474e5f640 1 --2- 172.21.15.1:0/1255863199 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fa45c065430 0x7fa45c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:35:54.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.455+0000 7fa465ffb640 1 --2- 172.21.15.1:0/1255863199 >> [v2:172.21.15.1:6810/2182416687,v1:172.21.15.1:6811/2182416687] conn(0x7fa45c06b1b0 0x7fa45c06d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:35:54.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.455+0000 7fa465ffb640 1 -- 172.21.15.1:0/1255863199 --> [v2:172.21.15.1:6810/2182416687,v1:172.21.15.1:6811/2182416687] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa45c06dc70 con 0x7fa45c06b1b0 2024-06-18T02:35:54.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.455+0000 7fa465ffb640 1 -- 172.21.15.1:0/1255863199 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa46c084710 con 0x7fa470064f20 2024-06-18T02:35:54.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.456+0000 7fa475e61640 1 --2- 172.21.15.1:0/1255863199 >> [v2:172.21.15.1:6810/2182416687,v1:172.21.15.1:6811/2182416687] conn(0x7fa45c06b1b0 0x7fa45c06d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:35:54.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.456+0000 7fa474e5f640 1 --2- 172.21.15.1:0/1255863199 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fa45c065430 0x7fa45c0678e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa460032430 tx=0x7fa460035040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:35:54.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.456+0000 7fa475e61640 1 --2- 172.21.15.1:0/1255863199 >> [v2:172.21.15.1:6810/2182416687,v1:172.21.15.1:6811/2182416687] conn(0x7fa45c06b1b0 0x7fa45c06d5c0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:35:54.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.457+0000 7fa465ffb640 1 -- 172.21.15.1:0/1255863199 <== osd.1 v2:172.21.15.1:6810/2182416687 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fa45c06dc70 con 0x7fa45c06b1b0 2024-06-18T02:35:54.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.471+0000 7fa4778c4640 1 -- 172.21.15.1:0/1255863199 --> [v2:172.21.15.1:6810/2182416687,v1:172.21.15.1:6811/2182416687] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa4700fc0d0 con 0x7fa45c06b1b0 2024-06-18T02:35:54.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.471+0000 7fa465ffb640 1 -- 172.21.15.1:0/1255863199 <== osd.1 v2:172.21.15.1:6810/2182416687 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa4700fc0d0 con 0x7fa45c06b1b0 2024-06-18T02:35:54.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.471+0000 7fa4778c4640 1 -- 172.21.15.1:0/1255863199 >> [v2:172.21.15.1:6810/2182416687,v1:172.21.15.1:6811/2182416687] conn(0x7fa45c06b1b0 msgr2=0x7fa45c06d5c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:35:54.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.471+0000 7fa4778c4640 1 --2- 172.21.15.1:0/1255863199 >> [v2:172.21.15.1:6810/2182416687,v1:172.21.15.1:6811/2182416687] conn(0x7fa45c06b1b0 0x7fa45c06d5c0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:54.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.471+0000 7fa4778c4640 1 -- 172.21.15.1:0/1255863199 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fa45c065430 msgr2=0x7fa45c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:35:54.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.471+0000 7fa4778c4640 1 --2- 172.21.15.1:0/1255863199 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fa45c065430 0x7fa45c0678e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa460032430 tx=0x7fa460035040 comp rx=0 tx=0).stop 2024-06-18T02:35:54.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.471+0000 7fa4778c4640 1 -- 172.21.15.1:0/1255863199 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa470064f20 msgr2=0x7fa470113010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:35:54.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.471+0000 7fa4778c4640 1 --2- 172.21.15.1:0/1255863199 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa470064f20 0x7fa470113010 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa46c00b790 tx=0x7fa46c0091d0 comp rx=0 tx=0).stop 2024-06-18T02:35:54.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.472+0000 7fa4778c4640 1 -- 172.21.15.1:0/1255863199 shutdown_connections 2024-06-18T02:35:54.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.472+0000 7fa4778c4640 1 --2- 172.21.15.1:0/1255863199 >> [v2:172.21.15.1:6810/2182416687,v1:172.21.15.1:6811/2182416687] conn(0x7fa45c06b1b0 0x7fa45c06d5c0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:54.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.472+0000 7fa4778c4640 1 --2- 172.21.15.1:0/1255863199 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fa45c065430 0x7fa45c0678e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:54.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.472+0000 7fa4778c4640 1 --2- 172.21.15.1:0/1255863199 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4700652f0 0x7fa470113550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:54.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.472+0000 7fa4778c4640 1 --2- 172.21.15.1:0/1255863199 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa470064f20 0x7fa470113010 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:54.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.472+0000 7fa4778c4640 1 -- 172.21.15.1:0/1255863199 >> 172.21.15.1:0/1255863199 conn(0x7fa470070a40 msgr2=0x7fa4700f9d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:35:54.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.472+0000 7fa4778c4640 1 -- 172.21.15.1:0/1255863199 shutdown_connections 2024-06-18T02:35:54.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:53.472+0000 7fa4778c4640 1 -- 172.21.15.1:0/1255863199 wait complete. 2024-06-18T02:35:54.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:53 smithi028 ceph-mon[35823]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:35:54.972 INFO:teuthology.orchestra.run.smithi001.stdout:60129542172 2024-06-18T02:35:54.972 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-06-18T02:35:55.126 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:35:56.065 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:55 smithi001 ceph-mon[29099]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:35:56.105 INFO:teuthology.orchestra.run.smithi001.stdout:81604378646 2024-06-18T02:35:56.105 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-06-18T02:35:56.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:55 smithi028 ceph-mon[35823]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:35:57.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:57.692+0000 7fe919f76640 1 -- 172.21.15.1:0/3712647199 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9140fb570 msgr2=0x7fe9140fb940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:35:57.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:57.692+0000 7fe919f76640 1 --2- 172.21.15.1:0/3712647199 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9140fb570 0x7fe9140fb940 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe9040048d0 tx=0x7fe904009da0 comp rx=0 tx=0).stop 2024-06-18T02:35:57.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:57.692+0000 7fe919f76640 1 -- 172.21.15.1:0/3712647199 shutdown_connections 2024-06-18T02:35:57.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:57.692+0000 7fe919f76640 1 --2- 172.21.15.1:0/3712647199 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9140fbe80 0x7fe9140fcd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:57.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:57.692+0000 7fe919f76640 1 --2- 172.21.15.1:0/3712647199 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9140fb570 0x7fe9140fb940 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:57.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:57.692+0000 7fe919f76640 1 -- 172.21.15.1:0/3712647199 >> 172.21.15.1:0/3712647199 conn(0x7fe914072110 msgr2=0x7fe914072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:35:57.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:57.692+0000 7fe919f76640 1 -- 172.21.15.1:0/3712647199 shutdown_connections 2024-06-18T02:35:57.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:57.692+0000 7fe919f76640 1 -- 172.21.15.1:0/3712647199 wait complete. 2024-06-18T02:35:57.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:57.693+0000 7fe919f76640 1 Processor -- start 2024-06-18T02:35:57.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:57.693+0000 7fe919f76640 1 -- start start 2024-06-18T02:35:57.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:57.693+0000 7fe919f76640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9140fb570 0x7fe91410cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:35:57.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:57.693+0000 7fe919f76640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9140fbe80 0x7fe91410d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:35:57.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:57.693+0000 7fe919f76640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe91410ae60 con 0x7fe9140fbe80 2024-06-18T02:35:57.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:57.693+0000 7fe919f76640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe91410afd0 con 0x7fe9140fb570 2024-06-18T02:35:57.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:57.694+0000 7fe912ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9140fbe80 0x7fe91410d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:35:57.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:57.694+0000 7fe912ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9140fbe80 0x7fe91410d2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40116/0 (socket says 172.21.15.1:40116) 2024-06-18T02:35:57.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:57.694+0000 7fe912ffd640 1 -- 172.21.15.1:0/4240123485 learned_addr learned my addr 172.21.15.1:0/4240123485 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:35:57.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:57.694+0000 7fe9137fe640 1 --2- 172.21.15.1:0/4240123485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9140fb570 0x7fe91410cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:35:57.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:57.694+0000 7fe912ffd640 1 -- 172.21.15.1:0/4240123485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9140fb570 msgr2=0x7fe91410cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:35:57.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:57.694+0000 7fe912ffd640 1 --2- 172.21.15.1:0/4240123485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9140fb570 0x7fe91410cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:57.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:57.694+0000 7fe912ffd640 1 -- 172.21.15.1:0/4240123485 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe904005040 con 0x7fe9140fbe80 2024-06-18T02:35:57.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:57.694+0000 7fe912ffd640 1 --2- 172.21.15.1:0/4240123485 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9140fbe80 0x7fe91410d2c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fe908002a80 tx=0x7fe908004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:35:57.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:57.694+0000 7fe8fbfff640 1 -- 172.21.15.1:0/4240123485 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe908007c30 con 0x7fe9140fbe80 2024-06-18T02:35:57.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:57.694+0000 7fe8fbfff640 1 -- 172.21.15.1:0/4240123485 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe90800f040 con 0x7fe9140fbe80 2024-06-18T02:35:57.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:57.694+0000 7fe8fbfff640 1 -- 172.21.15.1:0/4240123485 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe90800db70 con 0x7fe9140fbe80 2024-06-18T02:35:57.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:57.694+0000 7fe919f76640 1 -- 172.21.15.1:0/4240123485 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe91410b2b0 con 0x7fe9140fbe80 2024-06-18T02:35:57.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:57.695+0000 7fe919f76640 1 -- 172.21.15.1:0/4240123485 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe91410b7d0 con 0x7fe9140fbe80 2024-06-18T02:35:57.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:57.695+0000 7fe919f76640 1 -- 172.21.15.1:0/4240123485 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe9140fc920 con 0x7fe9140fbe80 2024-06-18T02:35:57.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:57.696+0000 7fe8fbfff640 1 -- 172.21.15.1:0/4240123485 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7fe90800dcd0 con 0x7fe9140fbe80 2024-06-18T02:35:57.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:57.697+0000 7fe8fbfff640 1 --2- 172.21.15.1:0/4240123485 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fe8fc065430 0x7fe8fc0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:35:57.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:57.697+0000 7fe8fbfff640 1 -- 172.21.15.1:0/4240123485 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fe908082220 con 0x7fe9140fbe80 2024-06-18T02:35:57.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:57.697+0000 7fe9137fe640 1 --2- 172.21.15.1:0/4240123485 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fe8fc065430 0x7fe8fc0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:35:57.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:57.697+0000 7fe8fbfff640 1 --2- 172.21.15.1:0/4240123485 >> [v2:172.21.15.28:6816/2643308264,v1:172.21.15.28:6817/2643308264] conn(0x7fe8fc06b1b0 0x7fe8fc06d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:35:57.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:57.697+0000 7fe8fbfff640 1 -- 172.21.15.1:0/4240123485 --> [v2:172.21.15.28:6816/2643308264,v1:172.21.15.28:6817/2643308264] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe8fc06dc70 con 0x7fe8fc06b1b0 2024-06-18T02:35:57.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:57.697+0000 7fe8fbfff640 1 -- 172.21.15.1:0/4240123485 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe908082600 con 0x7fe9140fbe80 2024-06-18T02:35:57.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:57.697+0000 7fe9137fe640 1 --2- 172.21.15.1:0/4240123485 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fe8fc065430 0x7fe8fc0678e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe904032430 tx=0x7fe904035040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:35:57.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:57.697+0000 7fe913fff640 1 --2- 172.21.15.1:0/4240123485 >> [v2:172.21.15.28:6816/2643308264,v1:172.21.15.28:6817/2643308264] conn(0x7fe8fc06b1b0 0x7fe8fc06d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:35:57.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:57.698+0000 7fe913fff640 1 --2- 172.21.15.1:0/4240123485 >> [v2:172.21.15.28:6816/2643308264,v1:172.21.15.28:6817/2643308264] conn(0x7fe8fc06b1b0 0x7fe8fc06d5c0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:35:57.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:57.699+0000 7fe8fbfff640 1 -- 172.21.15.1:0/4240123485 <== osd.5 v2:172.21.15.28:6816/2643308264 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fe8fc06dc70 con 0x7fe8fc06b1b0 2024-06-18T02:35:57.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:57.712+0000 7fe919f76640 1 -- 172.21.15.1:0/4240123485 --> [v2:172.21.15.28:6816/2643308264,v1:172.21.15.28:6817/2643308264] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe9140ff870 con 0x7fe8fc06b1b0 2024-06-18T02:35:57.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:57.712+0000 7fe8fbfff640 1 -- 172.21.15.1:0/4240123485 <== osd.5 v2:172.21.15.28:6816/2643308264 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fe9140ff870 con 0x7fe8fc06b1b0 2024-06-18T02:35:57.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:57.713+0000 7fe919f76640 1 -- 172.21.15.1:0/4240123485 >> [v2:172.21.15.28:6816/2643308264,v1:172.21.15.28:6817/2643308264] conn(0x7fe8fc06b1b0 msgr2=0x7fe8fc06d5c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:35:57.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:57.713+0000 7fe919f76640 1 --2- 172.21.15.1:0/4240123485 >> [v2:172.21.15.28:6816/2643308264,v1:172.21.15.28:6817/2643308264] conn(0x7fe8fc06b1b0 0x7fe8fc06d5c0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:57.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:57.713+0000 7fe919f76640 1 -- 172.21.15.1:0/4240123485 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fe8fc065430 msgr2=0x7fe8fc0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:35:57.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:57.713+0000 7fe919f76640 1 --2- 172.21.15.1:0/4240123485 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fe8fc065430 0x7fe8fc0678e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe904032430 tx=0x7fe904035040 comp rx=0 tx=0).stop 2024-06-18T02:35:57.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:57.713+0000 7fe919f76640 1 -- 172.21.15.1:0/4240123485 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9140fbe80 msgr2=0x7fe91410d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:35:57.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:57.713+0000 7fe919f76640 1 --2- 172.21.15.1:0/4240123485 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9140fbe80 0x7fe91410d2c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fe908002a80 tx=0x7fe908004600 comp rx=0 tx=0).stop 2024-06-18T02:35:57.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:57.713+0000 7fe919f76640 1 -- 172.21.15.1:0/4240123485 shutdown_connections 2024-06-18T02:35:57.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:57.713+0000 7fe919f76640 1 --2- 172.21.15.1:0/4240123485 >> [v2:172.21.15.28:6816/2643308264,v1:172.21.15.28:6817/2643308264] conn(0x7fe8fc06b1b0 0x7fe8fc06d5c0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:57.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:57.713+0000 7fe919f76640 1 --2- 172.21.15.1:0/4240123485 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fe8fc065430 0x7fe8fc0678e0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:57.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:57.713+0000 7fe919f76640 1 --2- 172.21.15.1:0/4240123485 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9140fbe80 0x7fe91410d2c0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:57.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:57.713+0000 7fe919f76640 1 --2- 172.21.15.1:0/4240123485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9140fb570 0x7fe91410cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:57.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:57.713+0000 7fe919f76640 1 -- 172.21.15.1:0/4240123485 >> 172.21.15.1:0/4240123485 conn(0x7fe914072110 msgr2=0x7fe9140fde90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:35:57.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:57.713+0000 7fe919f76640 1 -- 172.21.15.1:0/4240123485 shutdown_connections 2024-06-18T02:35:57.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:57.713+0000 7fe919f76640 1 -- 172.21.15.1:0/4240123485 wait complete. 2024-06-18T02:35:57.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:57 smithi001 ceph-mon[29099]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:35:58.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:57 smithi028 ceph-mon[35823]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:35:58.951 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:35:58.951 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:35:59.345 INFO:teuthology.orchestra.run.smithi001.stdout:167503724552 2024-06-18T02:35:59.345 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-06-18T02:35:59.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:58.830+0000 7f0b19ad3640 1 -- 172.21.15.1:0/556393466 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b140650a0 msgr2=0x7f0b14065470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:35:59.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:58.830+0000 7f0b19ad3640 1 --2- 172.21.15.1:0/556393466 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b140650a0 0x7f0b14065470 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f0b040048d0 tx=0x7f0b04009da0 comp rx=0 tx=0).stop 2024-06-18T02:35:59.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:58.830+0000 7f0b19ad3640 1 -- 172.21.15.1:0/556393466 shutdown_connections 2024-06-18T02:35:59.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:58.830+0000 7f0b19ad3640 1 --2- 172.21.15.1:0/556393466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b140fdeb0 0x7f0b140fe2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:59.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:58.830+0000 7f0b19ad3640 1 --2- 172.21.15.1:0/556393466 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b140650a0 0x7f0b14065470 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:59.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:58.830+0000 7f0b19ad3640 1 -- 172.21.15.1:0/556393466 >> 172.21.15.1:0/556393466 conn(0x7f0b14072110 msgr2=0x7f0b14072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:35:59.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:58.830+0000 7f0b19ad3640 1 -- 172.21.15.1:0/556393466 shutdown_connections 2024-06-18T02:35:59.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:58.830+0000 7f0b19ad3640 1 -- 172.21.15.1:0/556393466 wait complete. 2024-06-18T02:35:59.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:58.830+0000 7f0b19ad3640 1 Processor -- start 2024-06-18T02:35:59.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:58.830+0000 7f0b19ad3640 1 -- start start 2024-06-18T02:35:59.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:58.831+0000 7f0b19ad3640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b140650a0 0x7f0b14113080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:35:59.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:58.831+0000 7f0b19ad3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b140fdeb0 0x7f0b141135c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:35:59.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:58.831+0000 7f0b19ad3640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b14113c50 con 0x7f0b140650a0 2024-06-18T02:35:59.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:58.831+0000 7f0b19ad3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b1410d000 con 0x7f0b140fdeb0 2024-06-18T02:35:59.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:58.831+0000 7f0b137fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b140650a0 0x7f0b14113080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:35:59.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:58.831+0000 7f0b137fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b140650a0 0x7f0b14113080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40140/0 (socket says 172.21.15.1:40140) 2024-06-18T02:35:59.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:58.831+0000 7f0b137fe640 1 -- 172.21.15.1:0/834374589 learned_addr learned my addr 172.21.15.1:0/834374589 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:35:59.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:58.831+0000 7f0b12ffd640 1 --2- 172.21.15.1:0/834374589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b140fdeb0 0x7f0b141135c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:35:59.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:58.831+0000 7f0b137fe640 1 -- 172.21.15.1:0/834374589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b140fdeb0 msgr2=0x7f0b141135c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:35:59.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:58.831+0000 7f0b137fe640 1 --2- 172.21.15.1:0/834374589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b140fdeb0 0x7f0b141135c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:59.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:58.831+0000 7f0b137fe640 1 -- 172.21.15.1:0/834374589 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b04005040 con 0x7f0b140650a0 2024-06-18T02:35:59.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:58.831+0000 7f0b137fe640 1 --2- 172.21.15.1:0/834374589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b140650a0 0x7f0b14113080 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0b04003080 tx=0x7f0b04009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:35:59.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:58.831+0000 7f0afbfff640 1 -- 172.21.15.1:0/834374589 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b04038070 con 0x7f0b140650a0 2024-06-18T02:35:59.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:58.832+0000 7f0b19ad3640 1 -- 172.21.15.1:0/834374589 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b1410d280 con 0x7f0b140650a0 2024-06-18T02:35:59.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:58.832+0000 7f0afbfff640 1 -- 172.21.15.1:0/834374589 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0b04008ee0 con 0x7f0b140650a0 2024-06-18T02:35:59.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:58.832+0000 7f0afbfff640 1 -- 172.21.15.1:0/834374589 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b0403d870 con 0x7f0b140650a0 2024-06-18T02:35:59.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:58.832+0000 7f0b19ad3640 1 -- 172.21.15.1:0/834374589 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b1410d740 con 0x7f0b140650a0 2024-06-18T02:35:59.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:58.832+0000 7f0b19ad3640 1 -- 172.21.15.1:0/834374589 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0b1410dc20 con 0x7f0b140650a0 2024-06-18T02:35:59.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:58.833+0000 7f0afbfff640 1 -- 172.21.15.1:0/834374589 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7f0b04030020 con 0x7f0b140650a0 2024-06-18T02:35:59.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:58.833+0000 7f0afbfff640 1 --2- 172.21.15.1:0/834374589 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f0afc065430 0x7f0afc0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:35:59.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:58.834+0000 7f0afbfff640 1 -- 172.21.15.1:0/834374589 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f0b040a2240 con 0x7f0b140650a0 2024-06-18T02:35:59.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:58.834+0000 7f0b12ffd640 1 --2- 172.21.15.1:0/834374589 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f0afc065430 0x7f0afc0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:35:59.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:58.834+0000 7f0afbfff640 1 --2- 172.21.15.1:0/834374589 >> [v2:172.21.15.28:6800/664150046,v1:172.21.15.28:6801/664150046] conn(0x7f0afc06b1b0 0x7f0afc06d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:35:59.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:58.834+0000 7f0afbfff640 1 -- 172.21.15.1:0/834374589 --> [v2:172.21.15.28:6800/664150046,v1:172.21.15.28:6801/664150046] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0afc06dc70 con 0x7f0afc06b1b0 2024-06-18T02:35:59.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:58.834+0000 7f0afbfff640 1 -- 172.21.15.1:0/834374589 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0b040a4830 con 0x7f0b140650a0 2024-06-18T02:35:59.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:58.834+0000 7f0b13fff640 1 --2- 172.21.15.1:0/834374589 >> [v2:172.21.15.28:6800/664150046,v1:172.21.15.28:6801/664150046] conn(0x7f0afc06b1b0 0x7f0afc06d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:35:59.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:58.834+0000 7f0b12ffd640 1 --2- 172.21.15.1:0/834374589 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f0afc065430 0x7f0afc0678e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0b1410e940 tx=0x7f0b08008040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:35:59.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:58.834+0000 7f0b13fff640 1 --2- 172.21.15.1:0/834374589 >> [v2:172.21.15.28:6800/664150046,v1:172.21.15.28:6801/664150046] conn(0x7f0afc06b1b0 0x7f0afc06d5c0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:35:59.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:58.835+0000 7f0afbfff640 1 -- 172.21.15.1:0/834374589 <== osd.3 v2:172.21.15.28:6800/664150046 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f0afc06dc70 con 0x7f0afc06b1b0 2024-06-18T02:35:59.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:58.848+0000 7f0b19ad3640 1 -- 172.21.15.1:0/834374589 --> [v2:172.21.15.28:6800/664150046,v1:172.21.15.28:6801/664150046] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0b140fda50 con 0x7f0afc06b1b0 2024-06-18T02:35:59.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:58.849+0000 7f0afbfff640 1 -- 172.21.15.1:0/834374589 <== osd.3 v2:172.21.15.28:6800/664150046 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f0b140fda50 con 0x7f0afc06b1b0 2024-06-18T02:35:59.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:58.849+0000 7f0b19ad3640 1 -- 172.21.15.1:0/834374589 >> [v2:172.21.15.28:6800/664150046,v1:172.21.15.28:6801/664150046] conn(0x7f0afc06b1b0 msgr2=0x7f0afc06d5c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:35:59.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:58.849+0000 7f0b19ad3640 1 --2- 172.21.15.1:0/834374589 >> [v2:172.21.15.28:6800/664150046,v1:172.21.15.28:6801/664150046] conn(0x7f0afc06b1b0 0x7f0afc06d5c0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:59.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:58.849+0000 7f0b19ad3640 1 -- 172.21.15.1:0/834374589 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f0afc065430 msgr2=0x7f0afc0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:35:59.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:58.849+0000 7f0b19ad3640 1 --2- 172.21.15.1:0/834374589 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f0afc065430 0x7f0afc0678e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0b1410e940 tx=0x7f0b08008040 comp rx=0 tx=0).stop 2024-06-18T02:35:59.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:58.849+0000 7f0b19ad3640 1 -- 172.21.15.1:0/834374589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b140650a0 msgr2=0x7f0b14113080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:35:59.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:58.849+0000 7f0b19ad3640 1 --2- 172.21.15.1:0/834374589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b140650a0 0x7f0b14113080 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0b04003080 tx=0x7f0b04009da0 comp rx=0 tx=0).stop 2024-06-18T02:35:59.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:58.849+0000 7f0b19ad3640 1 -- 172.21.15.1:0/834374589 shutdown_connections 2024-06-18T02:35:59.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:58.849+0000 7f0b19ad3640 1 --2- 172.21.15.1:0/834374589 >> [v2:172.21.15.28:6800/664150046,v1:172.21.15.28:6801/664150046] conn(0x7f0afc06b1b0 0x7f0afc06d5c0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:59.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:58.849+0000 7f0b19ad3640 1 --2- 172.21.15.1:0/834374589 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f0afc065430 0x7f0afc0678e0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:59.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:58.849+0000 7f0b19ad3640 1 --2- 172.21.15.1:0/834374589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b140fdeb0 0x7f0b141135c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:59.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:58.849+0000 7f0b19ad3640 1 --2- 172.21.15.1:0/834374589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b140650a0 0x7f0b14113080 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:59.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:58.849+0000 7f0b19ad3640 1 -- 172.21.15.1:0/834374589 >> 172.21.15.1:0/834374589 conn(0x7f0b14072110 msgr2=0x7f0b14100fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:35:59.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:58.849+0000 7f0b19ad3640 1 -- 172.21.15.1:0/834374589 shutdown_connections 2024-06-18T02:35:59.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:58.849+0000 7f0b19ad3640 1 -- 172.21.15.1:0/834374589 wait complete. 2024-06-18T02:35:59.855 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:35:59 smithi001 ceph-mon[29099]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:35:59.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:59.862+0000 7f6e07b5a640 1 -- 172.21.15.1:0/3096889178 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e000fd190 msgr2=0x7f6e000fd560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:35:59.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:59.862+0000 7f6e07b5a640 1 --2- 172.21.15.1:0/3096889178 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e000fd190 0x7f6e000fd560 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc0048d0 tx=0x7f6dfc009da0 comp rx=0 tx=0).stop 2024-06-18T02:35:59.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:59.862+0000 7f6e07b5a640 1 -- 172.21.15.1:0/3096889178 shutdown_connections 2024-06-18T02:35:59.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:59.862+0000 7f6e07b5a640 1 --2- 172.21.15.1:0/3096889178 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e000fba50 0x7f6e000fbea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:59.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:59.862+0000 7f6e07b5a640 1 --2- 172.21.15.1:0/3096889178 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e000fd190 0x7f6e000fd560 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:59.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:59.862+0000 7f6e07b5a640 1 -- 172.21.15.1:0/3096889178 >> 172.21.15.1:0/3096889178 conn(0x7f6e00071860 msgr2=0x7f6e00071c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:35:59.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:59.862+0000 7f6e07b5a640 1 -- 172.21.15.1:0/3096889178 shutdown_connections 2024-06-18T02:35:59.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:59.862+0000 7f6e07b5a640 1 -- 172.21.15.1:0/3096889178 wait complete. 2024-06-18T02:35:59.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:59.863+0000 7f6e07b5a640 1 Processor -- start 2024-06-18T02:35:59.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:59.863+0000 7f6e07b5a640 1 -- start start 2024-06-18T02:35:59.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:59.863+0000 7f6e07b5a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e000fba50 0x7f6e00114130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:35:59.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:59.863+0000 7f6e07b5a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e000fd190 0x7f6e00114670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:35:59.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:59.863+0000 7f6e07b5a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e00114d50 con 0x7f6e000fba50 2024-06-18T02:35:59.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:59.863+0000 7f6e07b5a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e0010f2b0 con 0x7f6e000fd190 2024-06-18T02:35:59.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:59.863+0000 7f6e058f6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e000fba50 0x7f6e00114130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:35:59.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:59.863+0000 7f6e058f6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e000fba50 0x7f6e00114130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40162/0 (socket says 172.21.15.1:40162) 2024-06-18T02:35:59.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:59.863+0000 7f6e058f6640 1 -- 172.21.15.1:0/3508539385 learned_addr learned my addr 172.21.15.1:0/3508539385 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:35:59.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:59.864+0000 7f6e050f5640 1 --2- 172.21.15.1:0/3508539385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e000fd190 0x7f6e00114670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:35:59.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:59.864+0000 7f6e058f6640 1 -- 172.21.15.1:0/3508539385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e000fd190 msgr2=0x7f6e00114670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:35:59.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:59.864+0000 7f6e058f6640 1 --2- 172.21.15.1:0/3508539385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e000fd190 0x7f6e00114670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:59.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:59.864+0000 7f6e058f6640 1 -- 172.21.15.1:0/3508539385 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dfc005040 con 0x7f6e000fba50 2024-06-18T02:35:59.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:59.864+0000 7f6e058f6640 1 --2- 172.21.15.1:0/3508539385 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e000fba50 0x7f6e00114130 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc003080 tx=0x7f6dfc009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:35:59.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:59.864+0000 7f6df67fc640 1 -- 172.21.15.1:0/3508539385 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6dfc038070 con 0x7f6e000fba50 2024-06-18T02:35:59.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:59.864+0000 7f6e07b5a640 1 -- 172.21.15.1:0/3508539385 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e0010f530 con 0x7f6e000fba50 2024-06-18T02:35:59.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:59.864+0000 7f6df67fc640 1 -- 172.21.15.1:0/3508539385 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6dfc008ee0 con 0x7f6e000fba50 2024-06-18T02:35:59.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:59.864+0000 7f6df67fc640 1 -- 172.21.15.1:0/3508539385 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6dfc03d870 con 0x7f6e000fba50 2024-06-18T02:35:59.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:59.864+0000 7f6e07b5a640 1 -- 172.21.15.1:0/3508539385 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e0010f9f0 con 0x7f6e000fba50 2024-06-18T02:35:59.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:59.865+0000 7f6e07b5a640 1 -- 172.21.15.1:0/3508539385 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6e0010fe90 con 0x7f6e000fba50 2024-06-18T02:35:59.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:59.866+0000 7f6df67fc640 1 -- 172.21.15.1:0/3508539385 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7f6dfc030020 con 0x7f6e000fba50 2024-06-18T02:35:59.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:59.866+0000 7f6df67fc640 1 --2- 172.21.15.1:0/3508539385 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f6dec065430 0x7f6dec0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:35:59.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:59.866+0000 7f6df67fc640 1 -- 172.21.15.1:0/3508539385 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f6dfc0a2240 con 0x7f6e000fba50 2024-06-18T02:35:59.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:59.866+0000 7f6e050f5640 1 --2- 172.21.15.1:0/3508539385 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f6dec065430 0x7f6dec0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:35:59.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:59.866+0000 7f6df67fc640 1 --2- 172.21.15.1:0/3508539385 >> [v2:172.21.15.28:6808/276884153,v1:172.21.15.28:6809/276884153] conn(0x7f6dec06b1b0 0x7f6dec06d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:35:59.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:59.866+0000 7f6df67fc640 1 -- 172.21.15.1:0/3508539385 --> [v2:172.21.15.28:6808/276884153,v1:172.21.15.28:6809/276884153] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6dec06dc70 con 0x7f6dec06b1b0 2024-06-18T02:35:59.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:59.866+0000 7f6df67fc640 1 -- 172.21.15.1:0/3508539385 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6dfc0a4890 con 0x7f6e000fba50 2024-06-18T02:35:59.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:59.867+0000 7f6e060f7640 1 --2- 172.21.15.1:0/3508539385 >> [v2:172.21.15.28:6808/276884153,v1:172.21.15.28:6809/276884153] conn(0x7f6dec06b1b0 0x7f6dec06d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:35:59.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:59.867+0000 7f6e050f5640 1 --2- 172.21.15.1:0/3508539385 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f6dec065430 0x7f6dec0678e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6e00110b50 tx=0x7f6df0008040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:35:59.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:59.867+0000 7f6e060f7640 1 --2- 172.21.15.1:0/3508539385 >> [v2:172.21.15.28:6808/276884153,v1:172.21.15.28:6809/276884153] conn(0x7f6dec06b1b0 0x7f6dec06d5c0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:35:59.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:59.868+0000 7f6df67fc640 1 -- 172.21.15.1:0/3508539385 <== osd.4 v2:172.21.15.28:6808/276884153 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f6dec06dc70 con 0x7f6dec06b1b0 2024-06-18T02:35:59.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:59.894+0000 7f6e07b5a640 1 -- 172.21.15.1:0/3508539385 --> [v2:172.21.15.28:6808/276884153,v1:172.21.15.28:6809/276884153] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6e000fca60 con 0x7f6dec06b1b0 2024-06-18T02:35:59.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:59.895+0000 7f6df67fc640 1 -- 172.21.15.1:0/3508539385 <== osd.4 v2:172.21.15.28:6808/276884153 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f6e000fca60 con 0x7f6dec06b1b0 2024-06-18T02:35:59.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:59.895+0000 7f6e07b5a640 1 -- 172.21.15.1:0/3508539385 >> [v2:172.21.15.28:6808/276884153,v1:172.21.15.28:6809/276884153] conn(0x7f6dec06b1b0 msgr2=0x7f6dec06d5c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:35:59.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:59.895+0000 7f6e07b5a640 1 --2- 172.21.15.1:0/3508539385 >> [v2:172.21.15.28:6808/276884153,v1:172.21.15.28:6809/276884153] conn(0x7f6dec06b1b0 0x7f6dec06d5c0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:59.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:59.895+0000 7f6e07b5a640 1 -- 172.21.15.1:0/3508539385 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f6dec065430 msgr2=0x7f6dec0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:35:59.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:59.895+0000 7f6e07b5a640 1 --2- 172.21.15.1:0/3508539385 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f6dec065430 0x7f6dec0678e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6e00110b50 tx=0x7f6df0008040 comp rx=0 tx=0).stop 2024-06-18T02:35:59.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:59.895+0000 7f6e07b5a640 1 -- 172.21.15.1:0/3508539385 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e000fba50 msgr2=0x7f6e00114130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:35:59.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:59.895+0000 7f6e07b5a640 1 --2- 172.21.15.1:0/3508539385 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e000fba50 0x7f6e00114130 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc003080 tx=0x7f6dfc009da0 comp rx=0 tx=0).stop 2024-06-18T02:35:59.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:59.896+0000 7f6e07b5a640 1 -- 172.21.15.1:0/3508539385 shutdown_connections 2024-06-18T02:35:59.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:59.896+0000 7f6e07b5a640 1 --2- 172.21.15.1:0/3508539385 >> [v2:172.21.15.28:6808/276884153,v1:172.21.15.28:6809/276884153] conn(0x7f6dec06b1b0 0x7f6dec06d5c0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:59.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:59.896+0000 7f6e07b5a640 1 --2- 172.21.15.1:0/3508539385 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f6dec065430 0x7f6dec0678e0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:59.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:59.896+0000 7f6e07b5a640 1 --2- 172.21.15.1:0/3508539385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e000fd190 0x7f6e00114670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:59.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:59.896+0000 7f6e07b5a640 1 --2- 172.21.15.1:0/3508539385 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6e000fba50 0x7f6e00114130 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:35:59.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:59.896+0000 7f6e07b5a640 1 -- 172.21.15.1:0/3508539385 >> 172.21.15.1:0/3508539385 conn(0x7f6e00071860 msgr2=0x7f6e001002b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:35:59.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:59.896+0000 7f6e07b5a640 1 -- 172.21.15.1:0/3508539385 shutdown_connections 2024-06-18T02:35:59.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:35:59.896+0000 7f6e07b5a640 1 -- 172.21.15.1:0/3508539385 wait complete. 2024-06-18T02:36:00.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:35:59 smithi028 ceph-mon[35823]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:36:00.305 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:36:00.846 INFO:teuthology.orchestra.run.smithi001.stdout:111669149717 2024-06-18T02:36:00.846 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-06-18T02:36:01.210 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:36:01.970 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:36:01 smithi001 ceph-mon[29099]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:36:02.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:36:01 smithi028 ceph-mon[35823]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:36:02.785 INFO:teuthology.orchestra.run.smithi001.stdout:146028888079 2024-06-18T02:36:02.786 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-06-18T02:36:04.106 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:36:03 smithi001 ceph-mon[29099]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:36:04.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:36:03 smithi028 ceph-mon[35823]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:36:06.111 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:36:05 smithi001 ceph-mon[29099]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:36:06.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.199+0000 7fe099fa1640 1 -- 172.21.15.1:0/3984235562 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe094101040 msgr2=0x7fe094101410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:36:06.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.199+0000 7fe099fa1640 1 --2- 172.21.15.1:0/3984235562 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe094101040 0x7fe094101410 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fe0840048d0 tx=0x7fe084009da0 comp rx=0 tx=0).stop 2024-06-18T02:36:06.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.199+0000 7fe099fa1640 1 -- 172.21.15.1:0/3984235562 shutdown_connections 2024-06-18T02:36:06.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.199+0000 7fe099fa1640 1 --2- 172.21.15.1:0/3984235562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0941019e0 0x7fe0940fb070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:06.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.199+0000 7fe099fa1640 1 --2- 172.21.15.1:0/3984235562 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe094101040 0x7fe094101410 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:06.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.199+0000 7fe099fa1640 1 -- 172.21.15.1:0/3984235562 >> 172.21.15.1:0/3984235562 conn(0x7fe0940f6c10 msgr2=0x7fe0940f9020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:36:06.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.199+0000 7fe099fa1640 1 -- 172.21.15.1:0/3984235562 shutdown_connections 2024-06-18T02:36:06.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.199+0000 7fe099fa1640 1 -- 172.21.15.1:0/3984235562 wait complete. 2024-06-18T02:36:06.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.200+0000 7fe099fa1640 1 Processor -- start 2024-06-18T02:36:06.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.200+0000 7fe099fa1640 1 -- start start 2024-06-18T02:36:06.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.200+0000 7fe099fa1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe094101040 0x7fe094111d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:36:06.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.200+0000 7fe099fa1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0941019e0 0x7fe094112240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:36:06.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.200+0000 7fe099fa1640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe094112840 con 0x7fe094101040 2024-06-18T02:36:06.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.200+0000 7fe099fa1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0941129b0 con 0x7fe0941019e0 2024-06-18T02:36:06.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.200+0000 7fe098f9f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe094101040 0x7fe094111d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:36:06.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.200+0000 7fe098f9f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe094101040 0x7fe094111d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43630/0 (socket says 172.21.15.1:43630) 2024-06-18T02:36:06.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.200+0000 7fe098f9f640 1 -- 172.21.15.1:0/3228737361 learned_addr learned my addr 172.21.15.1:0/3228737361 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:36:06.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.201+0000 7fe098f9f640 1 -- 172.21.15.1:0/3228737361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0941019e0 msgr2=0x7fe094112240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:36:06.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.201+0000 7fe098f9f640 1 --2- 172.21.15.1:0/3228737361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0941019e0 0x7fe094112240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:06.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.201+0000 7fe098f9f640 1 -- 172.21.15.1:0/3228737361 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe084005040 con 0x7fe094101040 2024-06-18T02:36:06.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.201+0000 7fe098f9f640 1 --2- 172.21.15.1:0/3228737361 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe094101040 0x7fe094111d00 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fe084003080 tx=0x7fe084009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:36:06.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.201+0000 7fe0917fa640 1 -- 172.21.15.1:0/3228737361 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe084038070 con 0x7fe094101040 2024-06-18T02:36:06.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.201+0000 7fe0917fa640 1 -- 172.21.15.1:0/3228737361 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe084008ee0 con 0x7fe094101040 2024-06-18T02:36:06.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.201+0000 7fe099fa1640 1 -- 172.21.15.1:0/3228737361 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe09410bdd0 con 0x7fe094101040 2024-06-18T02:36:06.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.201+0000 7fe0917fa640 1 -- 172.21.15.1:0/3228737361 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe08403d870 con 0x7fe094101040 2024-06-18T02:36:06.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.201+0000 7fe099fa1640 1 -- 172.21.15.1:0/3228737361 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe09410c310 con 0x7fe094101040 2024-06-18T02:36:06.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.202+0000 7fe099fa1640 1 -- 172.21.15.1:0/3228737361 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0940fc7b0 con 0x7fe094101040 2024-06-18T02:36:06.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.202+0000 7fe0917fa640 1 -- 172.21.15.1:0/3228737361 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7fe084030020 con 0x7fe094101040 2024-06-18T02:36:06.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.203+0000 7fe0917fa640 1 --2- 172.21.15.1:0/3228737361 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fe07c065430 0x7fe07c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:36:06.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.203+0000 7fe0917fa640 1 -- 172.21.15.1:0/3228737361 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fe0840a2140 con 0x7fe094101040 2024-06-18T02:36:06.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.203+0000 7fe093fff640 1 --2- 172.21.15.1:0/3228737361 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fe07c065430 0x7fe07c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:36:06.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.203+0000 7fe093fff640 1 --2- 172.21.15.1:0/3228737361 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fe07c065430 0x7fe07c0678e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe09410d600 tx=0x7fe088007a10 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:36:06.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.207+0000 7fe0917fa640 1 -- 172.21.15.1:0/3228737361 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fe084070710 con 0x7fe094101040 2024-06-18T02:36:06.239 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:36:05 smithi028 ceph-mon[35823]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:36:06.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.382+0000 7fe099fa1640 1 -- 172.21.15.1:0/3228737361 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fe0940fa830 con 0x7fe094101040 2024-06-18T02:36:06.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.383+0000 7fe0917fa640 1 -- 172.21.15.1:0/3228737361 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fe084073d30 con 0x7fe094101040 2024-06-18T02:36:06.383 INFO:teuthology.orchestra.run.smithi001.stdout:167503724554 2024-06-18T02:36:06.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.385+0000 7fe099fa1640 1 -- 172.21.15.1:0/3228737361 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fe07c065430 msgr2=0x7fe07c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:36:06.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.385+0000 7fe099fa1640 1 --2- 172.21.15.1:0/3228737361 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fe07c065430 0x7fe07c0678e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe09410d600 tx=0x7fe088007a10 comp rx=0 tx=0).stop 2024-06-18T02:36:06.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.385+0000 7fe099fa1640 1 -- 172.21.15.1:0/3228737361 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe094101040 msgr2=0x7fe094111d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:36:06.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.385+0000 7fe099fa1640 1 --2- 172.21.15.1:0/3228737361 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe094101040 0x7fe094111d00 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fe084003080 tx=0x7fe084009da0 comp rx=0 tx=0).stop 2024-06-18T02:36:06.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.385+0000 7fe099fa1640 1 -- 172.21.15.1:0/3228737361 shutdown_connections 2024-06-18T02:36:06.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.385+0000 7fe099fa1640 1 --2- 172.21.15.1:0/3228737361 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fe07c065430 0x7fe07c0678e0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:06.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.385+0000 7fe099fa1640 1 --2- 172.21.15.1:0/3228737361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0941019e0 0x7fe094112240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:06.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.385+0000 7fe099fa1640 1 --2- 172.21.15.1:0/3228737361 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe094101040 0x7fe094111d00 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:06.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.385+0000 7fe099fa1640 1 -- 172.21.15.1:0/3228737361 >> 172.21.15.1:0/3228737361 conn(0x7fe0940f6c10 msgr2=0x7fe0940f7be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:36:06.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.385+0000 7fe099fa1640 1 -- 172.21.15.1:0/3228737361 shutdown_connections 2024-06-18T02:36:06.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.385+0000 7fe099fa1640 1 -- 172.21.15.1:0/3228737361 wait complete. 2024-06-18T02:36:06.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.849+0000 7fa832c63640 1 -- 172.21.15.1:0/1395613009 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa82c0fbe80 msgr2=0x7fa82c0fcd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:36:06.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.849+0000 7fa832c63640 1 --2- 172.21.15.1:0/1395613009 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa82c0fbe80 0x7fa82c0fcd80 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fa8200048d0 tx=0x7fa820009da0 comp rx=0 tx=0).stop 2024-06-18T02:36:06.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.849+0000 7fa832c63640 1 -- 172.21.15.1:0/1395613009 shutdown_connections 2024-06-18T02:36:06.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.849+0000 7fa832c63640 1 --2- 172.21.15.1:0/1395613009 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa82c0fbe80 0x7fa82c0fcd80 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:06.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.849+0000 7fa832c63640 1 --2- 172.21.15.1:0/1395613009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa82c0fb570 0x7fa82c0fb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:06.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.849+0000 7fa832c63640 1 -- 172.21.15.1:0/1395613009 >> 172.21.15.1:0/1395613009 conn(0x7fa82c072110 msgr2=0x7fa82c072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:36:06.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.849+0000 7fa832c63640 1 -- 172.21.15.1:0/1395613009 shutdown_connections 2024-06-18T02:36:06.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.849+0000 7fa832c63640 1 -- 172.21.15.1:0/1395613009 wait complete. 2024-06-18T02:36:06.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.849+0000 7fa832c63640 1 Processor -- start 2024-06-18T02:36:06.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.850+0000 7fa832c63640 1 -- start start 2024-06-18T02:36:06.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.850+0000 7fa832c63640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa82c0fb570 0x7fa82c111eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:36:06.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.850+0000 7fa832c63640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa82c0fbe80 0x7fa82c1123f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:36:06.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.850+0000 7fa832c63640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa82c1129f0 con 0x7fa82c0fbe80 2024-06-18T02:36:06.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.850+0000 7fa832c63640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa82c10be30 con 0x7fa82c0fb570 2024-06-18T02:36:06.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.850+0000 7fa82bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa82c0fbe80 0x7fa82c1123f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:36:06.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.850+0000 7fa82bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa82c0fbe80 0x7fa82c1123f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43656/0 (socket says 172.21.15.1:43656) 2024-06-18T02:36:06.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.850+0000 7fa82bfff640 1 -- 172.21.15.1:0/4275521785 learned_addr learned my addr 172.21.15.1:0/4275521785 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:36:06.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.850+0000 7fa8309ff640 1 --2- 172.21.15.1:0/4275521785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa82c0fb570 0x7fa82c111eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:36:06.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.850+0000 7fa82bfff640 1 -- 172.21.15.1:0/4275521785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa82c0fb570 msgr2=0x7fa82c111eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:36:06.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.850+0000 7fa82bfff640 1 --2- 172.21.15.1:0/4275521785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa82c0fb570 0x7fa82c111eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:06.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.850+0000 7fa82bfff640 1 -- 172.21.15.1:0/4275521785 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa820005040 con 0x7fa82c0fbe80 2024-06-18T02:36:06.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.850+0000 7fa82bfff640 1 --2- 172.21.15.1:0/4275521785 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa82c0fbe80 0x7fa82c1123f0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fa820003080 tx=0x7fa820009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:36:06.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.851+0000 7fa8297fa640 1 -- 172.21.15.1:0/4275521785 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa820038070 con 0x7fa82c0fbe80 2024-06-18T02:36:06.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.851+0000 7fa8297fa640 1 -- 172.21.15.1:0/4275521785 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa820032970 con 0x7fa82c0fbe80 2024-06-18T02:36:06.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.851+0000 7fa832c63640 1 -- 172.21.15.1:0/4275521785 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa82c10c0b0 con 0x7fa82c0fbe80 2024-06-18T02:36:06.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.851+0000 7fa8297fa640 1 -- 172.21.15.1:0/4275521785 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa82003d960 con 0x7fa82c0fbe80 2024-06-18T02:36:06.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.851+0000 7fa832c63640 1 -- 172.21.15.1:0/4275521785 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa82c10c4b0 con 0x7fa82c0fbe80 2024-06-18T02:36:06.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.852+0000 7fa832c63640 1 -- 172.21.15.1:0/4275521785 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa82c0fc920 con 0x7fa82c0fbe80 2024-06-18T02:36:06.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.852+0000 7fa8297fa640 1 -- 172.21.15.1:0/4275521785 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7fa820030020 con 0x7fa82c0fbe80 2024-06-18T02:36:06.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.853+0000 7fa8297fa640 1 --2- 172.21.15.1:0/4275521785 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fa814065430 0x7fa8140678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:36:06.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.853+0000 7fa8309ff640 1 --2- 172.21.15.1:0/4275521785 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fa814065430 0x7fa8140678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:36:06.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.853+0000 7fa8297fa640 1 -- 172.21.15.1:0/4275521785 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fa8200a2530 con 0x7fa82c0fbe80 2024-06-18T02:36:06.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.853+0000 7fa8309ff640 1 --2- 172.21.15.1:0/4275521785 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fa814065430 0x7fa8140678e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa81c009af0 tx=0x7fa81c00b040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:36:06.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:06.857+0000 7fa8297fa640 1 -- 172.21.15.1:0/4275521785 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fa8200709e0 con 0x7fa82c0fbe80 2024-06-18T02:36:06.907 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:36:07.037 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:36:06 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/3228737361' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-06-18T02:36:07.037 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:36:06 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:36:07.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:07.037+0000 7fa832c63640 1 -- 172.21.15.1:0/4275521785 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fa82c072c20 con 0x7fa82c0fbe80 2024-06-18T02:36:07.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:07.037+0000 7fa8297fa640 1 -- 172.21.15.1:0/4275521785 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa820074000 con 0x7fa82c0fbe80 2024-06-18T02:36:07.037 INFO:teuthology.orchestra.run.smithi001.stdout:81604378649 2024-06-18T02:36:07.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:07.039+0000 7fa832c63640 1 -- 172.21.15.1:0/4275521785 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fa814065430 msgr2=0x7fa8140678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:36:07.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:07.039+0000 7fa832c63640 1 --2- 172.21.15.1:0/4275521785 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fa814065430 0x7fa8140678e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa81c009af0 tx=0x7fa81c00b040 comp rx=0 tx=0).stop 2024-06-18T02:36:07.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:07.039+0000 7fa832c63640 1 -- 172.21.15.1:0/4275521785 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa82c0fbe80 msgr2=0x7fa82c1123f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:36:07.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:07.039+0000 7fa832c63640 1 --2- 172.21.15.1:0/4275521785 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa82c0fbe80 0x7fa82c1123f0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fa820003080 tx=0x7fa820009da0 comp rx=0 tx=0).stop 2024-06-18T02:36:07.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:07.040+0000 7fa832c63640 1 -- 172.21.15.1:0/4275521785 shutdown_connections 2024-06-18T02:36:07.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:07.040+0000 7fa832c63640 1 --2- 172.21.15.1:0/4275521785 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fa814065430 0x7fa8140678e0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:07.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:07.040+0000 7fa832c63640 1 --2- 172.21.15.1:0/4275521785 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa82c0fbe80 0x7fa82c1123f0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:07.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:07.040+0000 7fa832c63640 1 --2- 172.21.15.1:0/4275521785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa82c0fb570 0x7fa82c111eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:07.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:07.040+0000 7fa832c63640 1 -- 172.21.15.1:0/4275521785 >> 172.21.15.1:0/4275521785 conn(0x7fa82c072110 msgr2=0x7fa82c0ffea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:36:07.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:07.040+0000 7fa832c63640 1 -- 172.21.15.1:0/4275521785 shutdown_connections 2024-06-18T02:36:07.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:07.040+0000 7fa832c63640 1 -- 172.21.15.1:0/4275521785 wait complete. 2024-06-18T02:36:07.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:36:06 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/3228737361' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-06-18T02:36:07.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:36:06 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:36:07.429 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724552 got 167503724554 for osd.5 2024-06-18T02:36:07.429 DEBUG:teuthology.parallel:result is None 2024-06-18T02:36:07.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:07.599+0000 7f3176c9a640 1 -- 172.21.15.1:0/563494963 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31701022f0 msgr2=0x7f31701026c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:36:07.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:07.599+0000 7f3176c9a640 1 --2- 172.21.15.1:0/563494963 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31701022f0 0x7f31701026c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f31600048d0 tx=0x7f3160009da0 comp rx=0 tx=0).stop 2024-06-18T02:36:07.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:07.599+0000 7f3176c9a640 1 -- 172.21.15.1:0/563494963 shutdown_connections 2024-06-18T02:36:07.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:07.599+0000 7f3176c9a640 1 --2- 172.21.15.1:0/563494963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3170102c90 0x7f31700fc320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:07.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:07.599+0000 7f3176c9a640 1 --2- 172.21.15.1:0/563494963 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f31701022f0 0x7f31701026c0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:07.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:07.599+0000 7f3176c9a640 1 -- 172.21.15.1:0/563494963 >> 172.21.15.1:0/563494963 conn(0x7f31700f7f00 msgr2=0x7f31700fa310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:36:07.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:07.599+0000 7f3176c9a640 1 -- 172.21.15.1:0/563494963 shutdown_connections 2024-06-18T02:36:07.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:07.599+0000 7f3176c9a640 1 -- 172.21.15.1:0/563494963 wait complete. 2024-06-18T02:36:07.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:07.599+0000 7f3176c9a640 1 Processor -- start 2024-06-18T02:36:07.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:07.599+0000 7f3176c9a640 1 -- start start 2024-06-18T02:36:07.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:07.600+0000 7f3176c9a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31701022f0 0x7f3170113050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:36:07.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:07.600+0000 7f3176c9a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3170102c90 0x7f3170113590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:36:07.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:07.600+0000 7f3176c9a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3170113c20 con 0x7f3170102c90 2024-06-18T02:36:07.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:07.600+0000 7f3176c9a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f317010cfd0 con 0x7f31701022f0 2024-06-18T02:36:07.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:07.600+0000 7f316ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3170102c90 0x7f3170113590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:36:07.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:07.600+0000 7f316ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3170102c90 0x7f3170113590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43674/0 (socket says 172.21.15.1:43674) 2024-06-18T02:36:07.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:07.600+0000 7f316ffff640 1 -- 172.21.15.1:0/2828854460 learned_addr learned my addr 172.21.15.1:0/2828854460 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:36:07.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:07.600+0000 7f3174a36640 1 --2- 172.21.15.1:0/2828854460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31701022f0 0x7f3170113050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:36:07.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:07.600+0000 7f316ffff640 1 -- 172.21.15.1:0/2828854460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31701022f0 msgr2=0x7f3170113050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:36:07.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:07.600+0000 7f316ffff640 1 --2- 172.21.15.1:0/2828854460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31701022f0 0x7f3170113050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:07.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:07.600+0000 7f316ffff640 1 -- 172.21.15.1:0/2828854460 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3160005040 con 0x7f3170102c90 2024-06-18T02:36:07.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:07.600+0000 7f316ffff640 1 --2- 172.21.15.1:0/2828854460 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3170102c90 0x7f3170113590 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f316400b790 tx=0x7f31640091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:36:07.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:07.601+0000 7f316d7fa640 1 -- 172.21.15.1:0/2828854460 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f316400c700 con 0x7f3170102c90 2024-06-18T02:36:07.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:07.601+0000 7f3176c9a640 1 -- 172.21.15.1:0/2828854460 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f317010d2b0 con 0x7f3170102c90 2024-06-18T02:36:07.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:07.601+0000 7f316d7fa640 1 -- 172.21.15.1:0/2828854460 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f316400cd30 con 0x7f3170102c90 2024-06-18T02:36:07.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:07.601+0000 7f316d7fa640 1 -- 172.21.15.1:0/2828854460 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3164013370 con 0x7f3170102c90 2024-06-18T02:36:07.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:07.601+0000 7f3176c9a640 1 -- 172.21.15.1:0/2828854460 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f317010d7d0 con 0x7f3170102c90 2024-06-18T02:36:07.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:07.602+0000 7f3176c9a640 1 -- 172.21.15.1:0/2828854460 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31700fda60 con 0x7f3170102c90 2024-06-18T02:36:07.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:07.602+0000 7f316d7fa640 1 -- 172.21.15.1:0/2828854460 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7f3164012070 con 0x7f3170102c90 2024-06-18T02:36:07.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:07.603+0000 7f316d7fa640 1 --2- 172.21.15.1:0/2828854460 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f3158065430 0x7f31580678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:36:07.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:07.603+0000 7f316d7fa640 1 -- 172.21.15.1:0/2828854460 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f3164081e60 con 0x7f3170102c90 2024-06-18T02:36:07.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:07.603+0000 7f3174a36640 1 --2- 172.21.15.1:0/2828854460 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f3158065430 0x7f31580678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:36:07.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:07.603+0000 7f3174a36640 1 --2- 172.21.15.1:0/2828854460 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f3158065430 0x7f31580678e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3160009f80 tx=0x7f3160035040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:36:07.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:07.607+0000 7f316d7fa640 1 -- 172.21.15.1:0/2828854460 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f3164050270 con 0x7f3170102c90 2024-06-18T02:36:07.608 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:36:07.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:07.783+0000 7f3176c9a640 1 -- 172.21.15.1:0/2828854460 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f31700fbb00 con 0x7f3170102c90 2024-06-18T02:36:07.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:07.784+0000 7f316d7fa640 1 -- 172.21.15.1:0/2828854460 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3164053890 con 0x7f3170102c90 2024-06-18T02:36:07.783 INFO:teuthology.orchestra.run.smithi001.stdout:60129542175 2024-06-18T02:36:07.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:07.785+0000 7f3176c9a640 1 -- 172.21.15.1:0/2828854460 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f3158065430 msgr2=0x7f31580678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:36:07.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:07.785+0000 7f3176c9a640 1 --2- 172.21.15.1:0/2828854460 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f3158065430 0x7f31580678e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3160009f80 tx=0x7f3160035040 comp rx=0 tx=0).stop 2024-06-18T02:36:07.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:07.785+0000 7f3176c9a640 1 -- 172.21.15.1:0/2828854460 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3170102c90 msgr2=0x7f3170113590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:36:07.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:07.785+0000 7f3176c9a640 1 --2- 172.21.15.1:0/2828854460 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3170102c90 0x7f3170113590 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f316400b790 tx=0x7f31640091d0 comp rx=0 tx=0).stop 2024-06-18T02:36:07.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:07.786+0000 7f3176c9a640 1 -- 172.21.15.1:0/2828854460 shutdown_connections 2024-06-18T02:36:07.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:07.786+0000 7f3176c9a640 1 --2- 172.21.15.1:0/2828854460 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f3158065430 0x7f31580678e0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:07.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:07.786+0000 7f3176c9a640 1 --2- 172.21.15.1:0/2828854460 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3170102c90 0x7f3170113590 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:07.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:07.786+0000 7f3176c9a640 1 --2- 172.21.15.1:0/2828854460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31701022f0 0x7f3170113050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:07.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:07.786+0000 7f3176c9a640 1 -- 172.21.15.1:0/2828854460 >> 172.21.15.1:0/2828854460 conn(0x7f31700f7f00 msgr2=0x7f31700f8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:36:07.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:07.786+0000 7f3176c9a640 1 -- 172.21.15.1:0/2828854460 shutdown_connections 2024-06-18T02:36:07.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:07.786+0000 7f3176c9a640 1 -- 172.21.15.1:0/2828854460 wait complete. 2024-06-18T02:36:08.079 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:36:07 smithi001 ceph-mon[29099]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:36:08.079 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:36:07 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/4275521785' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-06-18T02:36:08.079 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:36:07 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/2828854460' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-06-18T02:36:08.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:36:07 smithi028 ceph-mon[35823]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:36:08.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:36:07 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/4275521785' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-06-18T02:36:08.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:36:07 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/2828854460' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-06-18T02:36:09.102 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542172 got 60129542175 for osd.1 2024-06-18T02:36:09.102 DEBUG:teuthology.parallel:result is None 2024-06-18T02:36:09.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:09.370+0000 7fd976631640 1 -- 172.21.15.1:0/760741882 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9700fd2a0 msgr2=0x7fd9700fd670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:36:09.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:09.370+0000 7fd976631640 1 --2- 172.21.15.1:0/760741882 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9700fd2a0 0x7fd9700fd670 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd960004900 tx=0x7fd960009da0 comp rx=0 tx=0).stop 2024-06-18T02:36:09.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:09.370+0000 7fd976631640 1 -- 172.21.15.1:0/760741882 shutdown_connections 2024-06-18T02:36:09.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:09.370+0000 7fd976631640 1 --2- 172.21.15.1:0/760741882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9700fdc40 0x7fd970106310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:09.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:09.370+0000 7fd976631640 1 --2- 172.21.15.1:0/760741882 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9700fd2a0 0x7fd9700fd670 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:09.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:09.370+0000 7fd976631640 1 -- 172.21.15.1:0/760741882 >> 172.21.15.1:0/760741882 conn(0x7fd9700f6d60 msgr2=0x7fd9700f9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:36:09.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:09.370+0000 7fd976631640 1 -- 172.21.15.1:0/760741882 shutdown_connections 2024-06-18T02:36:09.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:09.370+0000 7fd976631640 1 -- 172.21.15.1:0/760741882 wait complete. 2024-06-18T02:36:09.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:09.370+0000 7fd976631640 1 Processor -- start 2024-06-18T02:36:09.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:09.371+0000 7fd976631640 1 -- start start 2024-06-18T02:36:09.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:09.371+0000 7fd976631640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9700fd2a0 0x7fd9700fb160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:36:09.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:09.371+0000 7fd976631640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9700fdc40 0x7fd9700fb6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:36:09.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:09.371+0000 7fd976631640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9700fbbe0 con 0x7fd9700fdc40 2024-06-18T02:36:09.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:09.371+0000 7fd976631640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9700fbd50 con 0x7fd9700fd2a0 2024-06-18T02:36:09.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:09.371+0000 7fd96f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9700fdc40 0x7fd9700fb6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:36:09.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:09.371+0000 7fd96f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9700fdc40 0x7fd9700fb6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43694/0 (socket says 172.21.15.1:43694) 2024-06-18T02:36:09.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:09.371+0000 7fd96f7fe640 1 -- 172.21.15.1:0/1136452747 learned_addr learned my addr 172.21.15.1:0/1136452747 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:36:09.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:09.371+0000 7fd96ffff640 1 --2- 172.21.15.1:0/1136452747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9700fd2a0 0x7fd9700fb160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:36:09.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:09.372+0000 7fd96f7fe640 1 -- 172.21.15.1:0/1136452747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9700fd2a0 msgr2=0x7fd9700fb160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:36:09.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:09.372+0000 7fd96f7fe640 1 --2- 172.21.15.1:0/1136452747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9700fd2a0 0x7fd9700fb160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:09.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:09.372+0000 7fd96f7fe640 1 -- 172.21.15.1:0/1136452747 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd960005040 con 0x7fd9700fdc40 2024-06-18T02:36:09.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:09.372+0000 7fd96f7fe640 1 --2- 172.21.15.1:0/1136452747 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9700fdc40 0x7fd9700fb6a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd96400c790 tx=0x7fd9640043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:36:09.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:09.372+0000 7fd96cff9640 1 -- 172.21.15.1:0/1136452747 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd96400a7a0 con 0x7fd9700fdc40 2024-06-18T02:36:09.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:09.372+0000 7fd96cff9640 1 -- 172.21.15.1:0/1136452747 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd96400add0 con 0x7fd9700fdc40 2024-06-18T02:36:09.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:09.372+0000 7fd976631640 1 -- 172.21.15.1:0/1136452747 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9700fbfe0 con 0x7fd9700fdc40 2024-06-18T02:36:09.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:09.372+0000 7fd96cff9640 1 -- 172.21.15.1:0/1136452747 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd9640115e0 con 0x7fd9700fdc40 2024-06-18T02:36:09.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:09.372+0000 7fd976631640 1 -- 172.21.15.1:0/1136452747 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd970119740 con 0x7fd9700fdc40 2024-06-18T02:36:09.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:09.373+0000 7fd976631640 1 -- 172.21.15.1:0/1136452747 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd970066030 con 0x7fd9700fdc40 2024-06-18T02:36:09.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:09.374+0000 7fd96cff9640 1 -- 172.21.15.1:0/1136452747 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7fd964012400 con 0x7fd9700fdc40 2024-06-18T02:36:09.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:09.374+0000 7fd96cff9640 1 --2- 172.21.15.1:0/1136452747 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fd958065430 0x7fd9580678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:36:09.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:09.374+0000 7fd96cff9640 1 -- 172.21.15.1:0/1136452747 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fd9640825c0 con 0x7fd9700fdc40 2024-06-18T02:36:09.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:09.374+0000 7fd96ffff640 1 --2- 172.21.15.1:0/1136452747 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fd958065430 0x7fd9580678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:36:09.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:09.375+0000 7fd96ffff640 1 --2- 172.21.15.1:0/1136452747 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fd958065430 0x7fd9580678e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd960003080 tx=0x7fd960009da0 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:36:09.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:09.379+0000 7fd96cff9640 1 -- 172.21.15.1:0/1136452747 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fd964050c50 con 0x7fd9700fdc40 2024-06-18T02:36:09.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:09.554+0000 7fd976631640 1 -- 172.21.15.1:0/1136452747 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fd970109490 con 0x7fd9700fdc40 2024-06-18T02:36:09.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:09.555+0000 7fd96cff9640 1 -- 172.21.15.1:0/1136452747 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd964054270 con 0x7fd9700fdc40 2024-06-18T02:36:09.555 INFO:teuthology.orchestra.run.smithi001.stdout:38654705701 2024-06-18T02:36:09.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:09.557+0000 7fd976631640 1 -- 172.21.15.1:0/1136452747 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fd958065430 msgr2=0x7fd9580678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:36:09.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:09.557+0000 7fd976631640 1 --2- 172.21.15.1:0/1136452747 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fd958065430 0x7fd9580678e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd960003080 tx=0x7fd960009da0 comp rx=0 tx=0).stop 2024-06-18T02:36:09.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:09.557+0000 7fd976631640 1 -- 172.21.15.1:0/1136452747 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9700fdc40 msgr2=0x7fd9700fb6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:36:09.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:09.557+0000 7fd976631640 1 --2- 172.21.15.1:0/1136452747 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9700fdc40 0x7fd9700fb6a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd96400c790 tx=0x7fd9640043c0 comp rx=0 tx=0).stop 2024-06-18T02:36:09.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:09.557+0000 7fd976631640 1 -- 172.21.15.1:0/1136452747 shutdown_connections 2024-06-18T02:36:09.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:09.557+0000 7fd976631640 1 --2- 172.21.15.1:0/1136452747 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fd958065430 0x7fd9580678e0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:09.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:09.557+0000 7fd976631640 1 --2- 172.21.15.1:0/1136452747 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9700fdc40 0x7fd9700fb6a0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:09.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:09.557+0000 7fd976631640 1 --2- 172.21.15.1:0/1136452747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9700fd2a0 0x7fd9700fb160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:09.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:09.557+0000 7fd976631640 1 -- 172.21.15.1:0/1136452747 >> 172.21.15.1:0/1136452747 conn(0x7fd9700f6d60 msgr2=0x7fd9700f8280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:36:09.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:09.557+0000 7fd976631640 1 -- 172.21.15.1:0/1136452747 shutdown_connections 2024-06-18T02:36:09.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:09.557+0000 7fd976631640 1 -- 172.21.15.1:0/1136452747 wait complete. 2024-06-18T02:36:10.127 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:36:09 smithi001 ceph-mon[29099]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:36:10.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:36:09 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/1136452747' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-06-18T02:36:10.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:36:09 smithi028 ceph-mon[35823]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:36:10.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:36:09 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/1136452747' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-06-18T02:36:11.447 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705697 got 38654705701 for osd.0 2024-06-18T02:36:11.447 DEBUG:teuthology.parallel:result is None 2024-06-18T02:36:11.891 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378646 got 81604378649 for osd.2 2024-06-18T02:36:11.891 DEBUG:teuthology.parallel:result is None 2024-06-18T02:36:12.101 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:36:11 smithi001 ceph-mon[29099]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:36:12.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:36:11 smithi028 ceph-mon[35823]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:36:13.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.044+0000 7fd310530640 1 -- 172.21.15.1:0/3722671897 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3080fbaf0 msgr2=0x7fd3080fbf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:36:13.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.044+0000 7fd310530640 1 --2- 172.21.15.1:0/3722671897 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3080fbaf0 0x7fd3080fbf40 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc0048d0 tx=0x7fd2fc009da0 comp rx=0 tx=0).stop 2024-06-18T02:36:13.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.044+0000 7fd310530640 1 -- 172.21.15.1:0/3722671897 shutdown_connections 2024-06-18T02:36:13.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.044+0000 7fd310530640 1 --2- 172.21.15.1:0/3722671897 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3080fbaf0 0x7fd3080fbf40 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:13.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.044+0000 7fd310530640 1 --2- 172.21.15.1:0/3722671897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd308101250 0x7fd308101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:13.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.044+0000 7fd310530640 1 -- 172.21.15.1:0/3722671897 >> 172.21.15.1:0/3722671897 conn(0x7fd308070a40 msgr2=0x7fd308070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:36:13.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.044+0000 7fd310530640 1 -- 172.21.15.1:0/3722671897 shutdown_connections 2024-06-18T02:36:13.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.044+0000 7fd310530640 1 -- 172.21.15.1:0/3722671897 wait complete. 2024-06-18T02:36:13.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.045+0000 7fd310530640 1 Processor -- start 2024-06-18T02:36:13.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.045+0000 7fd310530640 1 -- start start 2024-06-18T02:36:13.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.045+0000 7fd310530640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3080fbaf0 0x7fd308111e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:36:13.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.045+0000 7fd310530640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd308101250 0x7fd3081123d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:36:13.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.045+0000 7fd310530640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3081129d0 con 0x7fd3080fbaf0 2024-06-18T02:36:13.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.045+0000 7fd310530640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd308112b40 con 0x7fd308101250 2024-06-18T02:36:13.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.045+0000 7fd30e2cc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3080fbaf0 0x7fd308111e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:36:13.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.045+0000 7fd30e2cc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3080fbaf0 0x7fd308111e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43708/0 (socket says 172.21.15.1:43708) 2024-06-18T02:36:13.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.045+0000 7fd30e2cc640 1 -- 172.21.15.1:0/1771675551 learned_addr learned my addr 172.21.15.1:0/1771675551 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:36:13.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.045+0000 7fd30dacb640 1 --2- 172.21.15.1:0/1771675551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd308101250 0x7fd3081123d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:36:13.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.046+0000 7fd30e2cc640 1 -- 172.21.15.1:0/1771675551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd308101250 msgr2=0x7fd3081123d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:36:13.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.046+0000 7fd30e2cc640 1 --2- 172.21.15.1:0/1771675551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd308101250 0x7fd3081123d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:13.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.046+0000 7fd30e2cc640 1 -- 172.21.15.1:0/1771675551 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2fc005040 con 0x7fd3080fbaf0 2024-06-18T02:36:13.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.046+0000 7fd30dacb640 1 --2- 172.21.15.1:0/1771675551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd308101250 0x7fd3081123d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-18T02:36:13.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.046+0000 7fd30e2cc640 1 --2- 172.21.15.1:0/1771675551 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3080fbaf0 0x7fd308111e90 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fd30400fe10 tx=0x7fd304004ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:36:13.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.046+0000 7fd2faffd640 1 -- 172.21.15.1:0/1771675551 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd304012460 con 0x7fd3080fbaf0 2024-06-18T02:36:13.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.046+0000 7fd310530640 1 -- 172.21.15.1:0/1771675551 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd30810c030 con 0x7fd3080fbaf0 2024-06-18T02:36:13.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.046+0000 7fd2faffd640 1 -- 172.21.15.1:0/1771675551 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd304012a90 con 0x7fd3080fbaf0 2024-06-18T02:36:13.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.046+0000 7fd2faffd640 1 -- 172.21.15.1:0/1771675551 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd30401aa80 con 0x7fd3080fbaf0 2024-06-18T02:36:13.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.046+0000 7fd310530640 1 -- 172.21.15.1:0/1771675551 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd30810c550 con 0x7fd3080fbaf0 2024-06-18T02:36:13.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.047+0000 7fd310530640 1 -- 172.21.15.1:0/1771675551 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3080fcb00 con 0x7fd3080fbaf0 2024-06-18T02:36:13.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.047+0000 7fd2faffd640 1 -- 172.21.15.1:0/1771675551 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7fd304012c00 con 0x7fd3080fbaf0 2024-06-18T02:36:13.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.048+0000 7fd2faffd640 1 --2- 172.21.15.1:0/1771675551 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fd2f4065430 0x7fd2f40678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:36:13.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.048+0000 7fd2faffd640 1 -- 172.21.15.1:0/1771675551 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fd304084f30 con 0x7fd3080fbaf0 2024-06-18T02:36:13.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.048+0000 7fd30dacb640 1 --2- 172.21.15.1:0/1771675551 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fd2f4065430 0x7fd2f40678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:36:13.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.048+0000 7fd30dacb640 1 --2- 172.21.15.1:0/1771675551 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fd2f4065430 0x7fd2f40678e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc000c00 tx=0x7fd2fc035040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:36:13.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.052+0000 7fd2faffd640 1 -- 172.21.15.1:0/1771675551 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fd304053460 con 0x7fd3080fbaf0 2024-06-18T02:36:13.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.229+0000 7fd310530640 1 -- 172.21.15.1:0/1771675551 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fd30810d5b0 con 0x7fd3080fbaf0 2024-06-18T02:36:13.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.229+0000 7fd2faffd640 1 -- 172.21.15.1:0/1771675551 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fd304056a80 con 0x7fd3080fbaf0 2024-06-18T02:36:13.229 INFO:teuthology.orchestra.run.smithi001.stdout:111669149719 2024-06-18T02:36:13.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.231+0000 7fd310530640 1 -- 172.21.15.1:0/1771675551 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fd2f4065430 msgr2=0x7fd2f40678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:36:13.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.231+0000 7fd310530640 1 --2- 172.21.15.1:0/1771675551 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fd2f4065430 0x7fd2f40678e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc000c00 tx=0x7fd2fc035040 comp rx=0 tx=0).stop 2024-06-18T02:36:13.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.231+0000 7fd310530640 1 -- 172.21.15.1:0/1771675551 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3080fbaf0 msgr2=0x7fd308111e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:36:13.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.231+0000 7fd310530640 1 --2- 172.21.15.1:0/1771675551 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3080fbaf0 0x7fd308111e90 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fd30400fe10 tx=0x7fd304004ee0 comp rx=0 tx=0).stop 2024-06-18T02:36:13.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.231+0000 7fd310530640 1 -- 172.21.15.1:0/1771675551 shutdown_connections 2024-06-18T02:36:13.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.231+0000 7fd310530640 1 --2- 172.21.15.1:0/1771675551 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fd2f4065430 0x7fd2f40678e0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:13.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.231+0000 7fd310530640 1 --2- 172.21.15.1:0/1771675551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd308101250 0x7fd3081123d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:13.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.231+0000 7fd310530640 1 --2- 172.21.15.1:0/1771675551 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3080fbaf0 0x7fd308111e90 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:13.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.231+0000 7fd310530640 1 -- 172.21.15.1:0/1771675551 >> 172.21.15.1:0/1771675551 conn(0x7fd308070a40 msgr2=0x7fd3080f8da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:36:13.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.231+0000 7fd310530640 1 -- 172.21.15.1:0/1771675551 shutdown_connections 2024-06-18T02:36:13.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.231+0000 7fd310530640 1 -- 172.21.15.1:0/1771675551 wait complete. 2024-06-18T02:36:13.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.242+0000 7f12ae9bd640 1 -- 172.21.15.1:0/1261222979 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12a80fead0 msgr2=0x7f12a80fa940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:36:13.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.242+0000 7f12ae9bd640 1 --2- 172.21.15.1:0/1261222979 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12a80fead0 0x7f12a80fa940 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f129c0048d0 tx=0x7f129c009da0 comp rx=0 tx=0).stop 2024-06-18T02:36:13.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.242+0000 7f12ae9bd640 1 -- 172.21.15.1:0/1261222979 shutdown_connections 2024-06-18T02:36:13.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.242+0000 7f12ae9bd640 1 --2- 172.21.15.1:0/1261222979 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12a80fead0 0x7f12a80fa940 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:13.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.242+0000 7f12ae9bd640 1 --2- 172.21.15.1:0/1261222979 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12a80fe1c0 0x7f12a80fe590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:13.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.242+0000 7f12ae9bd640 1 -- 172.21.15.1:0/1261222979 >> 172.21.15.1:0/1261222979 conn(0x7f12a8071860 msgr2=0x7f12a8071c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:36:13.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.242+0000 7f12ae9bd640 1 -- 172.21.15.1:0/1261222979 shutdown_connections 2024-06-18T02:36:13.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.242+0000 7f12ae9bd640 1 -- 172.21.15.1:0/1261222979 wait complete. 2024-06-18T02:36:13.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.243+0000 7f12ae9bd640 1 Processor -- start 2024-06-18T02:36:13.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.243+0000 7f12ae9bd640 1 -- start start 2024-06-18T02:36:13.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.243+0000 7f12ae9bd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12a80fe1c0 0x7f12a8115250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:36:13.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.243+0000 7f12ae9bd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12a80fead0 0x7f12a8115790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:36:13.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.243+0000 7f12ae9bd640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a8115e20 con 0x7f12a80fe1c0 2024-06-18T02:36:13.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.243+0000 7f12ae9bd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a810f1d0 con 0x7f12a80fead0 2024-06-18T02:36:13.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.243+0000 7f12a7fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12a80fe1c0 0x7f12a8115250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:36:13.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.243+0000 7f12a7fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12a80fe1c0 0x7f12a8115250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43730/0 (socket says 172.21.15.1:43730) 2024-06-18T02:36:13.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.243+0000 7f12a7fff640 1 -- 172.21.15.1:0/1875703906 learned_addr learned my addr 172.21.15.1:0/1875703906 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:36:13.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.243+0000 7f12a77fe640 1 --2- 172.21.15.1:0/1875703906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12a80fead0 0x7f12a8115790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:36:13.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.243+0000 7f12a7fff640 1 -- 172.21.15.1:0/1875703906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12a80fead0 msgr2=0x7f12a8115790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:36:13.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.243+0000 7f12a7fff640 1 --2- 172.21.15.1:0/1875703906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12a80fead0 0x7f12a8115790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:13.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.243+0000 7f12a7fff640 1 -- 172.21.15.1:0/1875703906 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f129c005040 con 0x7f12a80fe1c0 2024-06-18T02:36:13.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.244+0000 7f12a7fff640 1 --2- 172.21.15.1:0/1875703906 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12a80fe1c0 0x7f12a8115250 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f129800b790 tx=0x7f12980091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:36:13.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.244+0000 7f12a4ff9640 1 -- 172.21.15.1:0/1875703906 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f129800c700 con 0x7f12a80fe1c0 2024-06-18T02:36:13.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.244+0000 7f12a4ff9640 1 -- 172.21.15.1:0/1875703906 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f129800cd30 con 0x7f12a80fe1c0 2024-06-18T02:36:13.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.244+0000 7f12ae9bd640 1 -- 172.21.15.1:0/1875703906 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12a810f4b0 con 0x7f12a80fe1c0 2024-06-18T02:36:13.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.244+0000 7f12a4ff9640 1 -- 172.21.15.1:0/1875703906 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1298013370 con 0x7f12a80fe1c0 2024-06-18T02:36:13.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.244+0000 7f12ae9bd640 1 -- 172.21.15.1:0/1875703906 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12a810f9a0 con 0x7f12a80fe1c0 2024-06-18T02:36:13.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.245+0000 7f12ae9bd640 1 -- 172.21.15.1:0/1875703906 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12a8064ab0 con 0x7f12a80fe1c0 2024-06-18T02:36:13.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.246+0000 7f12a4ff9640 1 -- 172.21.15.1:0/1875703906 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7f1298012070 con 0x7f12a80fe1c0 2024-06-18T02:36:13.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.246+0000 7f12a4ff9640 1 --2- 172.21.15.1:0/1875703906 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f1290065430 0x7f12900678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:36:13.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.246+0000 7f12a4ff9640 1 -- 172.21.15.1:0/1875703906 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f1298081cc0 con 0x7f12a80fe1c0 2024-06-18T02:36:13.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.246+0000 7f12a77fe640 1 --2- 172.21.15.1:0/1875703906 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f1290065430 0x7f12900678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:36:13.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.247+0000 7f12a77fe640 1 --2- 172.21.15.1:0/1875703906 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f1290065430 0x7f12900678e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f129c009f80 tx=0x7f129c035040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:36:13.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.254+0000 7f12a4ff9640 1 -- 172.21.15.1:0/1875703906 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f12980501f0 con 0x7f12a80fe1c0 2024-06-18T02:36:13.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.432+0000 7f12ae9bd640 1 -- 172.21.15.1:0/1875703906 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f12a8072370 con 0x7f12a80fe1c0 2024-06-18T02:36:13.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.433+0000 7f12a4ff9640 1 -- 172.21.15.1:0/1875703906 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f1298053810 con 0x7f12a80fe1c0 2024-06-18T02:36:13.433 INFO:teuthology.orchestra.run.smithi001.stdout:146028888081 2024-06-18T02:36:13.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.435+0000 7f12ae9bd640 1 -- 172.21.15.1:0/1875703906 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f1290065430 msgr2=0x7f12900678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:36:13.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.435+0000 7f12ae9bd640 1 --2- 172.21.15.1:0/1875703906 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f1290065430 0x7f12900678e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f129c009f80 tx=0x7f129c035040 comp rx=0 tx=0).stop 2024-06-18T02:36:13.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.435+0000 7f12ae9bd640 1 -- 172.21.15.1:0/1875703906 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12a80fe1c0 msgr2=0x7f12a8115250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:36:13.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.435+0000 7f12ae9bd640 1 --2- 172.21.15.1:0/1875703906 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12a80fe1c0 0x7f12a8115250 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f129800b790 tx=0x7f12980091d0 comp rx=0 tx=0).stop 2024-06-18T02:36:13.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.435+0000 7f12ae9bd640 1 -- 172.21.15.1:0/1875703906 shutdown_connections 2024-06-18T02:36:13.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.435+0000 7f12ae9bd640 1 --2- 172.21.15.1:0/1875703906 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f1290065430 0x7f12900678e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:13.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.435+0000 7f12ae9bd640 1 --2- 172.21.15.1:0/1875703906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12a80fead0 0x7f12a8115790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:13.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.435+0000 7f12ae9bd640 1 --2- 172.21.15.1:0/1875703906 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f12a80fe1c0 0x7f12a8115250 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:13.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.435+0000 7f12ae9bd640 1 -- 172.21.15.1:0/1875703906 >> 172.21.15.1:0/1875703906 conn(0x7f12a8071860 msgr2=0x7f12a80f93d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:36:13.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.435+0000 7f12ae9bd640 1 -- 172.21.15.1:0/1875703906 shutdown_connections 2024-06-18T02:36:13.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:13.435+0000 7f12ae9bd640 1 -- 172.21.15.1:0/1875703906 wait complete. 2024-06-18T02:36:14.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:36:13 smithi001 ceph-mon[29099]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:36:14.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:36:13 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/1771675551' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-06-18T02:36:14.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:36:13 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/1875703906' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-06-18T02:36:14.160 INFO:tasks.cephadm.ceph_manager.ceph:need seq 146028888079 got 146028888081 for osd.4 2024-06-18T02:36:14.161 DEBUG:teuthology.parallel:result is None 2024-06-18T02:36:14.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:36:13 smithi028 ceph-mon[35823]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:36:14.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:36:13 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/1771675551' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-06-18T02:36:14.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:36:13 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/1875703906' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-06-18T02:36:14.527 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149717 got 111669149719 for osd.3 2024-06-18T02:36:14.527 DEBUG:teuthology.parallel:result is None 2024-06-18T02:36:14.527 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-06-18T02:36:14.527 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph pg dump --format=json 2024-06-18T02:36:14.799 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:36:16.127 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:36:15 smithi001 ceph-mon[29099]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:36:16.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:36:15 smithi028 ceph-mon[35823]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:36:16.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:16.471+0000 7f3854696640 1 -- 172.21.15.1:0/3265128880 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f384c101cb0 msgr2=0x7f384c0fb230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:36:16.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:16.471+0000 7f3854696640 1 --2- 172.21.15.1:0/3265128880 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f384c101cb0 0x7f384c0fb230 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f383c0048d0 tx=0x7f383c009da0 comp rx=0 tx=0).stop 2024-06-18T02:36:16.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:16.471+0000 7f3854696640 1 -- 172.21.15.1:0/3265128880 shutdown_connections 2024-06-18T02:36:16.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:16.471+0000 7f3854696640 1 --2- 172.21.15.1:0/3265128880 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f384c101cb0 0x7f384c0fb230 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:16.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:16.471+0000 7f3854696640 1 --2- 172.21.15.1:0/3265128880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f384c101310 0x7f384c1016e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:16.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:16.471+0000 7f3854696640 1 -- 172.21.15.1:0/3265128880 >> 172.21.15.1:0/3265128880 conn(0x7f384c0f6d90 msgr2=0x7f384c0f91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:36:16.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:16.472+0000 7f3854696640 1 -- 172.21.15.1:0/3265128880 shutdown_connections 2024-06-18T02:36:16.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:16.472+0000 7f3854696640 1 -- 172.21.15.1:0/3265128880 wait complete. 2024-06-18T02:36:16.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:16.472+0000 7f3854696640 1 Processor -- start 2024-06-18T02:36:16.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:16.472+0000 7f3854696640 1 -- start start 2024-06-18T02:36:16.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:16.472+0000 7f3854696640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f384c101310 0x7f384c113100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:36:16.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:16.472+0000 7f3854696640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f384c101cb0 0x7f384c113640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:36:16.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:16.472+0000 7f3854696640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f384c113c40 con 0x7f384c101cb0 2024-06-18T02:36:16.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:16.472+0000 7f3854696640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f384c10d080 con 0x7f384c101310 2024-06-18T02:36:16.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:16.472+0000 7f3851c31640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f384c101cb0 0x7f384c113640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:36:16.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:16.472+0000 7f3851c31640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f384c101cb0 0x7f384c113640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45954/0 (socket says 172.21.15.1:45954) 2024-06-18T02:36:16.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:16.472+0000 7f3851c31640 1 -- 172.21.15.1:0/4014714969 learned_addr learned my addr 172.21.15.1:0/4014714969 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:36:16.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:16.473+0000 7f3852432640 1 --2- 172.21.15.1:0/4014714969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f384c101310 0x7f384c113100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:36:16.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:16.473+0000 7f3851c31640 1 -- 172.21.15.1:0/4014714969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f384c101310 msgr2=0x7f384c113100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:36:16.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:16.473+0000 7f3851c31640 1 --2- 172.21.15.1:0/4014714969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f384c101310 0x7f384c113100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:16.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:16.473+0000 7f3851c31640 1 -- 172.21.15.1:0/4014714969 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f383c005040 con 0x7f384c101cb0 2024-06-18T02:36:16.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:16.473+0000 7f3852432640 1 --2- 172.21.15.1:0/4014714969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f384c101310 0x7f384c113100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-18T02:36:16.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:16.473+0000 7f3851c31640 1 --2- 172.21.15.1:0/4014714969 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f384c101cb0 0x7f384c113640 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f383c032430 tx=0x7f383c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:36:16.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:16.473+0000 7f3842ffd640 1 -- 172.21.15.1:0/4014714969 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f383c038070 con 0x7f384c101cb0 2024-06-18T02:36:16.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:16.473+0000 7f3842ffd640 1 -- 172.21.15.1:0/4014714969 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f383c008ee0 con 0x7f384c101cb0 2024-06-18T02:36:16.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:16.473+0000 7f3854696640 1 -- 172.21.15.1:0/4014714969 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f384c10d300 con 0x7f384c101cb0 2024-06-18T02:36:16.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:16.473+0000 7f3842ffd640 1 -- 172.21.15.1:0/4014714969 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f383c03d7a0 con 0x7f384c101cb0 2024-06-18T02:36:16.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:16.473+0000 7f3854696640 1 -- 172.21.15.1:0/4014714969 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f384c10d840 con 0x7f384c101cb0 2024-06-18T02:36:16.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:16.474+0000 7f3854696640 1 -- 172.21.15.1:0/4014714969 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f384c0fa9f0 con 0x7f384c101cb0 2024-06-18T02:36:16.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:16.474+0000 7f3842ffd640 1 -- 172.21.15.1:0/4014714969 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7f383c030020 con 0x7f384c101cb0 2024-06-18T02:36:16.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:16.475+0000 7f3842ffd640 1 --2- 172.21.15.1:0/4014714969 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f3838065430 0x7f38380678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:36:16.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:16.475+0000 7f3842ffd640 1 -- 172.21.15.1:0/4014714969 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f383c0a2a50 con 0x7f384c101cb0 2024-06-18T02:36:16.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:16.475+0000 7f3852432640 1 --2- 172.21.15.1:0/4014714969 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f3838065430 0x7f38380678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:36:16.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:16.475+0000 7f3852432640 1 --2- 172.21.15.1:0/4014714969 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f3838065430 0x7f38380678e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f38480041d0 tx=0x7f3848009040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:36:16.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:16.479+0000 7f3842ffd640 1 -- 172.21.15.1:0/4014714969 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f383c071040 con 0x7f384c101cb0 2024-06-18T02:36:16.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:16.650+0000 7f3854696640 1 -- 172.21.15.1:0/4014714969 --> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f384c04b480 con 0x7f3838065430 2024-06-18T02:36:16.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:16.651+0000 7f3842ffd640 1 -- 172.21.15.1:0/4014714969 <== mgr.14216 v2:172.21.15.1:6800/729353917 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+28048 (secure 0 0 0) 0x7f384c04b480 con 0x7f3838065430 2024-06-18T02:36:16.651 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:36:16.651 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-06-18T02:36:16.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:16.653+0000 7f3854696640 1 -- 172.21.15.1:0/4014714969 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f3838065430 msgr2=0x7f38380678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:36:16.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:16.653+0000 7f3854696640 1 --2- 172.21.15.1:0/4014714969 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f3838065430 0x7f38380678e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f38480041d0 tx=0x7f3848009040 comp rx=0 tx=0).stop 2024-06-18T02:36:16.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:16.653+0000 7f3854696640 1 -- 172.21.15.1:0/4014714969 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f384c101cb0 msgr2=0x7f384c113640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:36:16.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:16.653+0000 7f3854696640 1 --2- 172.21.15.1:0/4014714969 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f384c101cb0 0x7f384c113640 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f383c032430 tx=0x7f383c009da0 comp rx=0 tx=0).stop 2024-06-18T02:36:16.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:16.653+0000 7f3854696640 1 -- 172.21.15.1:0/4014714969 shutdown_connections 2024-06-18T02:36:16.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:16.653+0000 7f3854696640 1 --2- 172.21.15.1:0/4014714969 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f3838065430 0x7f38380678e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:16.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:16.653+0000 7f3854696640 1 --2- 172.21.15.1:0/4014714969 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f384c101cb0 0x7f384c113640 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:16.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:16.653+0000 7f3854696640 1 --2- 172.21.15.1:0/4014714969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f384c101310 0x7f384c113100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:16.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:16.653+0000 7f3854696640 1 -- 172.21.15.1:0/4014714969 >> 172.21.15.1:0/4014714969 conn(0x7f384c0f6d90 msgr2=0x7f384c0f7df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:36:16.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:16.653+0000 7f3854696640 1 -- 172.21.15.1:0/4014714969 shutdown_connections 2024-06-18T02:36:16.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:16.653+0000 7f3854696640 1 -- 172.21.15.1:0/4014714969 wait complete. 2024-06-18T02:36:17.397 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":153,"stamp":"2024-06-18T02:36:16.566117+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1788028,"kb_used_data":3876,"kb_used_omap":9,"kb_used_meta":1784054,"kb_avail":560682884,"statfs":{"total":575970213888,"available":574139273216,"internally_reserved":0,"allocated":3969024,"data_stored":2606262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1826871312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002644"},"pg_stats":[{"pgid":"1.0","version":"23'117","reported_seq":189,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-06-18T02:35:21.964676+0000","last_change":"2024-06-18T02:34:57.898142+0000","last_active":"2024-06-18T02:35:21.964676+0000","last_peered":"2024-06-18T02:35:21.964676+0000","last_clean":"2024-06-18T02:35:21.964676+0000","last_became_active":"2024-06-18T02:34:57.897834+0000","last_became_peered":"2024-06-18T02:34:57.897834+0000","last_unstale":"2024-06-18T02:35:21.964676+0000","last_undegraded":"2024-06-18T02:35:21.964676+0000","last_fullsized":"2024-06-18T02:35:21.964676+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-18T02:34:11.222115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-18T02:34:11.222115+0000","last_clean_scrub_stamp":"2024-06-18T02:34:11.222115+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-19T03:44:43.429336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":39,"seq":167503724556,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297712,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447440,"statfs":{"total":95995035648,"available":95690178560,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.374}]}]},{"osd":4,"up_from":34,"seq":146028888082,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297716,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447436,"statfs":{"total":95995035648,"available":95690174464,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jun 18 02:35:59 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.63}]},{"osd":1,"last update":"Tue Jun 18 02:35:59 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.60799999999999998}]},{"osd":2,"last update":"Tue Jun 18 02:35:59 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.57099999999999995}]},{"osd":3,"last update":"Tue Jun 18 02:35:59 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.442},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.41999999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46600000000000003}]}]},{"osd":3,"up_from":26,"seq":111669149720,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298296,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446856,"statfs":{"total":95995035648,"available":95689580544,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jun 18 02:35:33 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.622},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.59399999999999997}]},{"osd":1,"last update":"Tue Jun 18 02:35:33 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.64900000000000002}]},{"osd":2,"last update":"Tue Jun 18 02:35:33 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.68200000000000005}]},{"osd":4,"last update":"Tue Jun 18 02:35:58 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.375,"15min":0.375},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"last":0.47199999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46100000000000002}]}]},{"osd":2,"up_from":19,"seq":81604378651,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297712,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447440,"statfs":{"total":95995035648,"available":95690178560,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jun 18 02:36:13 2024","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.35599999999999998,"15min":0.36199999999999999},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.46899999999999997,"5min":0.623,"15min":0.623},"last":0.24199999999999999},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.378,"15min":0.38300000000000001},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.28100000000000003}]},{"osd":1,"last update":"Tue Jun 18 02:36:13 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.38700000000000001,"15min":0.39100000000000001},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.39100000000000001,"15min":0.39600000000000002},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.26600000000000001}]},{"osd":3,"last update":"Tue Jun 18 02:35:33 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.58899999999999997}]},{"osd":4,"last update":"Tue Jun 18 02:36:01 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":8.3819999999999997,"5min":8.3819999999999997,"15min":8.3819999999999997},"last":0.52300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]}]},{"osd":1,"up_from":14,"seq":60129542177,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298292,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446860,"statfs":{"total":95995035648,"available":95689584640,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jun 18 02:35:50 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.308,"15min":0.30399999999999999},"min":{"1min":0.246,"5min":0.216,"15min":0.216},"max":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.31900000000000001,"15min":0.314},"min":{"1min":0.26800000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.46200000000000002,"5min":0.51400000000000001,"15min":0.51400000000000001},"last":0.32800000000000001}]},{"osd":2,"last update":"Tue Jun 18 02:35:12 2024","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.505,"5min":0.505,"15min":0.505},"last":0.30199999999999999}]},{"osd":3,"last update":"Tue Jun 18 02:35:33 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.59899999999999998}]},{"osd":4,"last update":"Tue Jun 18 02:35:59 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.57099999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.627}]}]},{"osd":0,"up_from":9,"seq":38654705702,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298300,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446852,"statfs":{"total":95995035648,"available":95689576448,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jun 18 02:35:49 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.318,"15min":0.312},"min":{"1min":0.247,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.46700000000000003,"5min":0.51000000000000001,"15min":0.51000000000000001},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.316,"15min":0.308},"min":{"1min":0.23499999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.46000000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"last":0.46200000000000002}]},{"osd":2,"last update":"Tue Jun 18 02:35:12 2024","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.371,"15min":0.371},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.5,"5min":0.5,"15min":0.5},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"last":0.55600000000000005}]},{"osd":3,"last update":"Tue Jun 18 02:35:34 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.80500000000000005}]},{"osd":4,"last update":"Tue Jun 18 02:35:59 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.81200000000000006}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-18T02:36:17.398 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph pg dump --format=json 2024-06-18T02:36:17.665 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:36:18.112 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:36:17 smithi001 ceph-mon[29099]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:36:18.112 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:36:17 smithi001 ceph-mon[29099]: from='client.14428 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-18T02:36:18.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:36:17 smithi028 ceph-mon[35823]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:36:18.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:36:17 smithi028 ceph-mon[35823]: from='client.14428 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-18T02:36:19.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:19.251+0000 7fe4f21ca640 1 -- 172.21.15.1:0/1413677997 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe4ec102c60 msgr2=0x7fe4ec0fc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:36:19.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:19.251+0000 7fe4f21ca640 1 --2- 172.21.15.1:0/1413677997 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe4ec102c60 0x7fe4ec0fc2f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fe4e00048d0 tx=0x7fe4e0009da0 comp rx=0 tx=0).stop 2024-06-18T02:36:19.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:19.252+0000 7fe4f21ca640 1 -- 172.21.15.1:0/1413677997 shutdown_connections 2024-06-18T02:36:19.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:19.252+0000 7fe4f21ca640 1 --2- 172.21.15.1:0/1413677997 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe4ec102c60 0x7fe4ec0fc2f0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:19.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:19.252+0000 7fe4f21ca640 1 --2- 172.21.15.1:0/1413677997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4ec1022c0 0x7fe4ec102690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:19.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:19.252+0000 7fe4f21ca640 1 -- 172.21.15.1:0/1413677997 >> 172.21.15.1:0/1413677997 conn(0x7fe4ec0f7e50 msgr2=0x7fe4ec0fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:36:19.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:19.252+0000 7fe4f21ca640 1 -- 172.21.15.1:0/1413677997 shutdown_connections 2024-06-18T02:36:19.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:19.252+0000 7fe4f21ca640 1 -- 172.21.15.1:0/1413677997 wait complete. 2024-06-18T02:36:19.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:19.252+0000 7fe4f21ca640 1 Processor -- start 2024-06-18T02:36:19.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:19.252+0000 7fe4f21ca640 1 -- start start 2024-06-18T02:36:19.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:19.253+0000 7fe4f21ca640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4ec1022c0 0x7fe4ec06f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:36:19.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:19.253+0000 7fe4f21ca640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe4ec102c60 0x7fe4ec06fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:36:19.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:19.253+0000 7fe4f21ca640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4ec0701f0 con 0x7fe4ec102c60 2024-06-18T02:36:19.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:19.253+0000 7fe4f21ca640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4ec070360 con 0x7fe4ec1022c0 2024-06-18T02:36:19.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:19.253+0000 7fe4eaffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe4ec102c60 0x7fe4ec06fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:36:19.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:19.253+0000 7fe4eb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4ec1022c0 0x7fe4ec06f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:36:19.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:19.253+0000 7fe4eb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4ec1022c0 0x7fe4ec06f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.1:35546/0 (socket says 172.21.15.1:35546) 2024-06-18T02:36:19.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:19.253+0000 7fe4eb7fe640 1 -- 172.21.15.1:0/3966748042 learned_addr learned my addr 172.21.15.1:0/3966748042 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:36:19.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:19.253+0000 7fe4eaffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe4ec102c60 0x7fe4ec06fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45960/0 (socket says 172.21.15.1:45960) 2024-06-18T02:36:19.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:19.253+0000 7fe4eb7fe640 1 -- 172.21.15.1:0/3966748042 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe4ec102c60 msgr2=0x7fe4ec06fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:36:19.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:19.253+0000 7fe4eb7fe640 1 --2- 172.21.15.1:0/3966748042 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe4ec102c60 0x7fe4ec06fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:19.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:19.253+0000 7fe4eb7fe640 1 -- 172.21.15.1:0/3966748042 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4e0005040 con 0x7fe4ec1022c0 2024-06-18T02:36:19.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:19.253+0000 7fe4eb7fe640 1 --2- 172.21.15.1:0/3966748042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4ec1022c0 0x7fe4ec06f770 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe4dc00ece0 tx=0x7fe4dc00b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:36:19.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:19.253+0000 7fe4d3fff640 1 -- 172.21.15.1:0/3966748042 <== mon.1 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe4dc00ee40 con 0x7fe4ec1022c0 2024-06-18T02:36:19.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:19.253+0000 7fe4d3fff640 1 -- 172.21.15.1:0/3966748042 <== mon.1 v2:172.21.15.28:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe4dc00f040 con 0x7fe4ec1022c0 2024-06-18T02:36:19.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:19.253+0000 7fe4f21ca640 1 -- 172.21.15.1:0/3966748042 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4ec0705f0 con 0x7fe4ec1022c0 2024-06-18T02:36:19.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:19.253+0000 7fe4d3fff640 1 -- 172.21.15.1:0/3966748042 <== mon.1 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe4dc009670 con 0x7fe4ec1022c0 2024-06-18T02:36:19.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:19.253+0000 7fe4f21ca640 1 -- 172.21.15.1:0/3966748042 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4ec119a70 con 0x7fe4ec1022c0 2024-06-18T02:36:19.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:19.254+0000 7fe4f21ca640 1 -- 172.21.15.1:0/3966748042 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4ec0fda30 con 0x7fe4ec1022c0 2024-06-18T02:36:19.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:19.255+0000 7fe4d3fff640 1 -- 172.21.15.1:0/3966748042 <== mon.1 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7fe4dc007890 con 0x7fe4ec1022c0 2024-06-18T02:36:19.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:19.255+0000 7fe4d3fff640 1 --2- 172.21.15.1:0/3966748042 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fe4d40653d0 0x7fe4d4067880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:36:19.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:19.255+0000 7fe4d3fff640 1 -- 172.21.15.1:0/3966748042 <== mon.1 v2:172.21.15.28:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fe4dc082bd0 con 0x7fe4ec1022c0 2024-06-18T02:36:19.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:19.255+0000 7fe4eaffd640 1 --2- 172.21.15.1:0/3966748042 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fe4d40653d0 0x7fe4d4067880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:36:19.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:19.256+0000 7fe4eaffd640 1 --2- 172.21.15.1:0/3966748042 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fe4d40653d0 0x7fe4d4067880 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe4e00048a0 tx=0x7fe4e0035040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:36:19.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:19.260+0000 7fe4d3fff640 1 -- 172.21.15.1:0/3966748042 <== mon.1 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fe4dc050fe0 con 0x7fe4ec1022c0 2024-06-18T02:36:19.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:19.430+0000 7fe4f21ca640 1 -- 172.21.15.1:0/3966748042 --> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe4ec070e60 con 0x7fe4d40653d0 2024-06-18T02:36:19.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:19.432+0000 7fe4d3fff640 1 -- 172.21.15.1:0/3966748042 <== mgr.14216 v2:172.21.15.1:6800/729353917 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+28183 (secure 0 0 0) 0x7fe4ec070e60 con 0x7fe4d40653d0 2024-06-18T02:36:19.432 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:36:19.432 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-06-18T02:36:19.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:19.434+0000 7fe4f21ca640 1 -- 172.21.15.1:0/3966748042 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fe4d40653d0 msgr2=0x7fe4d4067880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:36:19.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:19.434+0000 7fe4f21ca640 1 --2- 172.21.15.1:0/3966748042 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fe4d40653d0 0x7fe4d4067880 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe4e00048a0 tx=0x7fe4e0035040 comp rx=0 tx=0).stop 2024-06-18T02:36:19.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:19.434+0000 7fe4f21ca640 1 -- 172.21.15.1:0/3966748042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4ec1022c0 msgr2=0x7fe4ec06f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:36:19.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:19.434+0000 7fe4f21ca640 1 --2- 172.21.15.1:0/3966748042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4ec1022c0 0x7fe4ec06f770 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe4dc00ece0 tx=0x7fe4dc00b040 comp rx=0 tx=0).stop 2024-06-18T02:36:19.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:19.434+0000 7fe4f21ca640 1 -- 172.21.15.1:0/3966748042 shutdown_connections 2024-06-18T02:36:19.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:19.434+0000 7fe4f21ca640 1 --2- 172.21.15.1:0/3966748042 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fe4d40653d0 0x7fe4d4067880 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:19.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:19.434+0000 7fe4f21ca640 1 --2- 172.21.15.1:0/3966748042 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe4ec102c60 0x7fe4ec06fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:19.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:19.434+0000 7fe4f21ca640 1 --2- 172.21.15.1:0/3966748042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4ec1022c0 0x7fe4ec06f770 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:19.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:19.434+0000 7fe4f21ca640 1 -- 172.21.15.1:0/3966748042 >> 172.21.15.1:0/3966748042 conn(0x7fe4ec0f7e50 msgr2=0x7fe4ec0f8db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:36:19.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:19.434+0000 7fe4f21ca640 1 -- 172.21.15.1:0/3966748042 shutdown_connections 2024-06-18T02:36:19.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:19.434+0000 7fe4f21ca640 1 -- 172.21.15.1:0/3966748042 wait complete. 2024-06-18T02:36:20.023 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:36:19 smithi001 ceph-mon[29099]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:36:20.049 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":154,"stamp":"2024-06-18T02:36:18.566459+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1788028,"kb_used_data":3876,"kb_used_omap":9,"kb_used_meta":1784054,"kb_avail":560682884,"statfs":{"total":575970213888,"available":574139273216,"internally_reserved":0,"allocated":3969024,"data_stored":2606262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1826871312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002492"},"pg_stats":[{"pgid":"1.0","version":"23'117","reported_seq":189,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-06-18T02:35:21.964676+0000","last_change":"2024-06-18T02:34:57.898142+0000","last_active":"2024-06-18T02:35:21.964676+0000","last_peered":"2024-06-18T02:35:21.964676+0000","last_clean":"2024-06-18T02:35:21.964676+0000","last_became_active":"2024-06-18T02:34:57.897834+0000","last_became_peered":"2024-06-18T02:34:57.897834+0000","last_unstale":"2024-06-18T02:35:21.964676+0000","last_undegraded":"2024-06-18T02:35:21.964676+0000","last_fullsized":"2024-06-18T02:35:21.964676+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-18T02:34:11.222115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-18T02:34:11.222115+0000","last_clean_scrub_stamp":"2024-06-18T02:34:11.222115+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-19T03:44:43.429336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":39,"seq":167503724557,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297712,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447440,"statfs":{"total":95995035648,"available":95690178560,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51600000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48499999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32600000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.307}]}]},{"osd":4,"up_from":34,"seq":146028888083,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297716,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447436,"statfs":{"total":95995035648,"available":95690174464,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jun 18 02:35:59 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.46400000000000002}]},{"osd":1,"last update":"Tue Jun 18 02:35:59 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.40500000000000003}]},{"osd":2,"last update":"Tue Jun 18 02:35:59 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.41799999999999998}]},{"osd":3,"last update":"Tue Jun 18 02:35:59 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.32700000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38800000000000001}]}]},{"osd":3,"up_from":26,"seq":111669149720,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298296,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446856,"statfs":{"total":95995035648,"available":95689580544,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jun 18 02:35:33 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.622},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.59399999999999997}]},{"osd":1,"last update":"Tue Jun 18 02:35:33 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.64900000000000002}]},{"osd":2,"last update":"Tue Jun 18 02:35:33 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.68200000000000005}]},{"osd":4,"last update":"Tue Jun 18 02:35:58 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.375,"15min":0.375},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"last":0.47199999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46100000000000002}]}]},{"osd":2,"up_from":19,"seq":81604378651,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297712,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447440,"statfs":{"total":95995035648,"available":95690178560,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jun 18 02:36:13 2024","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.35599999999999998,"15min":0.36199999999999999},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.46899999999999997,"5min":0.623,"15min":0.623},"last":0.24199999999999999},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.378,"15min":0.38300000000000001},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.28100000000000003}]},{"osd":1,"last update":"Tue Jun 18 02:36:13 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.38700000000000001,"15min":0.39100000000000001},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.39100000000000001,"15min":0.39600000000000002},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.26600000000000001}]},{"osd":3,"last update":"Tue Jun 18 02:35:33 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.58899999999999997}]},{"osd":4,"last update":"Tue Jun 18 02:36:01 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":8.3819999999999997,"5min":8.3819999999999997,"15min":8.3819999999999997},"last":0.52300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]}]},{"osd":1,"up_from":14,"seq":60129542177,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298292,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446860,"statfs":{"total":95995035648,"available":95689584640,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jun 18 02:35:50 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.308,"15min":0.30399999999999999},"min":{"1min":0.246,"5min":0.216,"15min":0.216},"max":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.31900000000000001,"15min":0.314},"min":{"1min":0.26800000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.46200000000000002,"5min":0.51400000000000001,"15min":0.51400000000000001},"last":0.32800000000000001}]},{"osd":2,"last update":"Tue Jun 18 02:35:12 2024","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.505,"5min":0.505,"15min":0.505},"last":0.30199999999999999}]},{"osd":3,"last update":"Tue Jun 18 02:35:33 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.59899999999999998}]},{"osd":4,"last update":"Tue Jun 18 02:35:59 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.57099999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.627}]}]},{"osd":0,"up_from":9,"seq":38654705703,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298300,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446852,"statfs":{"total":95995035648,"available":95689576448,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jun 18 02:35:49 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.318,"15min":0.312},"min":{"1min":0.247,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.46700000000000003,"5min":0.51000000000000001,"15min":0.51000000000000001},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.316,"15min":0.308},"min":{"1min":0.23499999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.46000000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"last":0.32000000000000001}]},{"osd":2,"last update":"Tue Jun 18 02:36:14 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.36699999999999999,"15min":0.36899999999999999},"min":{"1min":0.25700000000000001,"5min":0.22,"15min":0.22},"max":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.377,"5min":0.38600000000000001,"15min":0.38800000000000001},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.35199999999999998}]},{"osd":3,"last update":"Tue Jun 18 02:35:34 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.56399999999999995}]},{"osd":4,"last update":"Tue Jun 18 02:35:59 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.52100000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-18T02:36:20.050 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-06-18T02:36:20.050 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-06-18T02:36:20.050 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-06-18T02:36:20.050 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T02:36:20.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:36:19 smithi028 ceph-mon[35823]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:36:20.322 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:36:21.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:36:20 smithi001 ceph-mon[29099]: from='client.24255 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-18T02:36:21.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:36:20 smithi028 ceph-mon[35823]: from='client.24255 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-18T02:36:21.954 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:36:21 smithi001 ceph-mon[29099]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:36:21.954 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:36:21 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:36:22.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:22.154+0000 7f1038c0f640 1 -- 172.21.15.1:0/1402238100 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1034101ba0 msgr2=0x7f10340fb230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:36:22.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:22.154+0000 7f1038c0f640 1 --2- 172.21.15.1:0/1402238100 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1034101ba0 0x7f10340fb230 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f101c0048d0 tx=0x7f101c009da0 comp rx=0 tx=0).stop 2024-06-18T02:36:22.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:22.154+0000 7f1038c0f640 1 -- 172.21.15.1:0/1402238100 shutdown_connections 2024-06-18T02:36:22.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:22.154+0000 7f1038c0f640 1 --2- 172.21.15.1:0/1402238100 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1034101ba0 0x7f10340fb230 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:22.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:22.154+0000 7f1038c0f640 1 --2- 172.21.15.1:0/1402238100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1034101200 0x7f10341015d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:22.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:22.154+0000 7f1038c0f640 1 -- 172.21.15.1:0/1402238100 >> 172.21.15.1:0/1402238100 conn(0x7f10340f6df0 msgr2=0x7f10340f9200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:36:22.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:22.154+0000 7f1038c0f640 1 -- 172.21.15.1:0/1402238100 shutdown_connections 2024-06-18T02:36:22.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:22.154+0000 7f1038c0f640 1 -- 172.21.15.1:0/1402238100 wait complete. 2024-06-18T02:36:22.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:22.155+0000 7f1038c0f640 1 Processor -- start 2024-06-18T02:36:22.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:22.155+0000 7f1038c0f640 1 -- start start 2024-06-18T02:36:22.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:22.155+0000 7f1038c0f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1034101200 0x7f1034115230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:36:22.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:22.155+0000 7f1038c0f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1034101ba0 0x7f1034115770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:36:22.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:22.155+0000 7f1038c0f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1034115d70 con 0x7f1034101200 2024-06-18T02:36:22.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:22.155+0000 7f1038c0f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f103410f1b0 con 0x7f1034101ba0 2024-06-18T02:36:22.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:22.155+0000 7f103259c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1034101200 0x7f1034115230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:36:22.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:22.155+0000 7f103259c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1034101200 0x7f1034115230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45980/0 (socket says 172.21.15.1:45980) 2024-06-18T02:36:22.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:22.155+0000 7f103259c640 1 -- 172.21.15.1:0/3889134323 learned_addr learned my addr 172.21.15.1:0/3889134323 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:36:22.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:22.155+0000 7f1031d9b640 1 --2- 172.21.15.1:0/3889134323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1034101ba0 0x7f1034115770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:36:22.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:22.155+0000 7f103259c640 1 -- 172.21.15.1:0/3889134323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1034101ba0 msgr2=0x7f1034115770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:36:22.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:22.155+0000 7f103259c640 1 --2- 172.21.15.1:0/3889134323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1034101ba0 0x7f1034115770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:22.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:22.155+0000 7f103259c640 1 -- 172.21.15.1:0/3889134323 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f101c005040 con 0x7f1034101200 2024-06-18T02:36:22.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:22.156+0000 7f103259c640 1 --2- 172.21.15.1:0/3889134323 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1034101200 0x7f1034115230 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f102400db80 tx=0x7f1024004f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:36:22.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:22.156+0000 7f102affd640 1 -- 172.21.15.1:0/3889134323 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f102400b8d0 con 0x7f1034101200 2024-06-18T02:36:22.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:22.156+0000 7f1038c0f640 1 -- 172.21.15.1:0/3889134323 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f103410f490 con 0x7f1034101200 2024-06-18T02:36:22.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:22.156+0000 7f102affd640 1 -- 172.21.15.1:0/3889134323 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f102400f040 con 0x7f1034101200 2024-06-18T02:36:22.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:22.156+0000 7f102affd640 1 -- 172.21.15.1:0/3889134323 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1024012390 con 0x7f1034101200 2024-06-18T02:36:22.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:22.156+0000 7f1038c0f640 1 -- 172.21.15.1:0/3889134323 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f103410f980 con 0x7f1034101200 2024-06-18T02:36:22.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:22.157+0000 7f1038c0f640 1 -- 172.21.15.1:0/3889134323 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10340fc970 con 0x7f1034101200 2024-06-18T02:36:22.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:22.157+0000 7f102affd640 1 -- 172.21.15.1:0/3889134323 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7f102400ba30 con 0x7f1034101200 2024-06-18T02:36:22.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:22.158+0000 7f102affd640 1 --2- 172.21.15.1:0/3889134323 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f1020065430 0x7f10200678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:36:22.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:22.158+0000 7f102affd640 1 -- 172.21.15.1:0/3889134323 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f1024082380 con 0x7f1034101200 2024-06-18T02:36:22.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:22.158+0000 7f1031d9b640 1 --2- 172.21.15.1:0/3889134323 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f1020065430 0x7f10200678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:36:22.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:22.158+0000 7f1031d9b640 1 --2- 172.21.15.1:0/3889134323 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f1020065430 0x7f10200678e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f101c009f80 tx=0x7f101c035040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:36:22.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:22.162+0000 7f102affd640 1 -- 172.21.15.1:0/3889134323 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f1024050830 con 0x7f1034101200 2024-06-18T02:36:22.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:36:21 smithi028 ceph-mon[35823]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:36:22.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:36:21 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:36:22.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:22.369+0000 7f1038c0f640 1 -- 172.21.15.1:0/3889134323 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f103404b480 con 0x7f1034101200 2024-06-18T02:36:22.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:22.370+0000 7f102affd640 1 -- 172.21.15.1:0/3889134323 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f1024053e50 con 0x7f1034101200 2024-06-18T02:36:22.370 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:36:22.371 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-18T02:36:22.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:22.372+0000 7f1038c0f640 1 -- 172.21.15.1:0/3889134323 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f1020065430 msgr2=0x7f10200678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:36:22.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:22.372+0000 7f1038c0f640 1 --2- 172.21.15.1:0/3889134323 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f1020065430 0x7f10200678e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f101c009f80 tx=0x7f101c035040 comp rx=0 tx=0).stop 2024-06-18T02:36:22.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:22.372+0000 7f1038c0f640 1 -- 172.21.15.1:0/3889134323 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1034101200 msgr2=0x7f1034115230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:36:22.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:22.372+0000 7f1038c0f640 1 --2- 172.21.15.1:0/3889134323 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1034101200 0x7f1034115230 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f102400db80 tx=0x7f1024004f00 comp rx=0 tx=0).stop 2024-06-18T02:36:22.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:22.372+0000 7f1038c0f640 1 -- 172.21.15.1:0/3889134323 shutdown_connections 2024-06-18T02:36:22.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:22.372+0000 7f1038c0f640 1 --2- 172.21.15.1:0/3889134323 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f1020065430 0x7f10200678e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:22.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:22.372+0000 7f1038c0f640 1 --2- 172.21.15.1:0/3889134323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1034101ba0 0x7f1034115770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:22.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:22.372+0000 7f1038c0f640 1 --2- 172.21.15.1:0/3889134323 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1034101200 0x7f1034115230 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:22.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:22.372+0000 7f1038c0f640 1 -- 172.21.15.1:0/3889134323 >> 172.21.15.1:0/3889134323 conn(0x7f10340f6df0 msgr2=0x7f10340f7dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:36:22.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:22.372+0000 7f1038c0f640 1 -- 172.21.15.1:0/3889134323 shutdown_connections 2024-06-18T02:36:22.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:22.372+0000 7f1038c0f640 1 -- 172.21.15.1:0/3889134323 wait complete. 2024-06-18T02:36:22.952 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:36:22 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/3889134323' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:36:23.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:36:22 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/3889134323' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:36:24.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:36:23 smithi001 ceph-mon[29099]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:36:24.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:36:23 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:36:24.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:36:23 smithi028 ceph-mon[35823]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:36:24.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:36:23 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:36:25.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:36:24 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:36:25.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:36:24 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T02:36:25.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:36:24 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:36:25.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:36:24 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:36:25.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:36:24 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T02:36:25.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:36:24 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:36:25.982 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T02:36:26.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:36:25 smithi028 ceph-mon[35823]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:36:26.258 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:36:26.377 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:36:25 smithi001 ceph-mon[29099]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:36:27.996 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:36:27 smithi001 ceph-mon[29099]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:36:28.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:28.078+0000 7f805eb19640 1 -- 172.21.15.1:0/122985736 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8058101ba0 msgr2=0x7f80580fb230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:36:28.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:28.078+0000 7f805eb19640 1 --2- 172.21.15.1:0/122985736 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8058101ba0 0x7f80580fb230 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f804c0048d0 tx=0x7f804c009da0 comp rx=0 tx=0).stop 2024-06-18T02:36:28.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:28.078+0000 7f805eb19640 1 -- 172.21.15.1:0/122985736 shutdown_connections 2024-06-18T02:36:28.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:28.078+0000 7f805eb19640 1 --2- 172.21.15.1:0/122985736 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8058101ba0 0x7f80580fb230 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:28.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:28.078+0000 7f805eb19640 1 --2- 172.21.15.1:0/122985736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8058101200 0x7f80581015d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:28.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:28.078+0000 7f805eb19640 1 -- 172.21.15.1:0/122985736 >> 172.21.15.1:0/122985736 conn(0x7f80580f6df0 msgr2=0x7f80580f9200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:36:28.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:28.078+0000 7f805eb19640 1 -- 172.21.15.1:0/122985736 shutdown_connections 2024-06-18T02:36:28.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:28.078+0000 7f805eb19640 1 -- 172.21.15.1:0/122985736 wait complete. 2024-06-18T02:36:28.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:28.078+0000 7f805eb19640 1 Processor -- start 2024-06-18T02:36:28.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:28.078+0000 7f805eb19640 1 -- start start 2024-06-18T02:36:28.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:28.079+0000 7f805eb19640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8058101200 0x7f8058111f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:36:28.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:28.079+0000 7f805eb19640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8058101ba0 0x7f80581124d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:36:28.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:28.079+0000 7f805eb19640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8058112b60 con 0x7f8058101ba0 2024-06-18T02:36:28.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:28.079+0000 7f805eb19640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f805810bf10 con 0x7f8058101200 2024-06-18T02:36:28.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:28.079+0000 7f8057fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8058101ba0 0x7f80581124d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:36:28.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:28.079+0000 7f8057fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8058101ba0 0x7f80581124d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42956/0 (socket says 172.21.15.1:42956) 2024-06-18T02:36:28.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:28.079+0000 7f8057fff640 1 -- 172.21.15.1:0/256882161 learned_addr learned my addr 172.21.15.1:0/256882161 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:36:28.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:28.079+0000 7f805c8b5640 1 --2- 172.21.15.1:0/256882161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8058101200 0x7f8058111f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:36:28.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:28.079+0000 7f8057fff640 1 -- 172.21.15.1:0/256882161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8058101200 msgr2=0x7f8058111f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:36:28.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:28.079+0000 7f8057fff640 1 --2- 172.21.15.1:0/256882161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8058101200 0x7f8058111f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:28.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:28.079+0000 7f8057fff640 1 -- 172.21.15.1:0/256882161 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f804c005040 con 0x7f8058101ba0 2024-06-18T02:36:28.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:28.079+0000 7f8057fff640 1 --2- 172.21.15.1:0/256882161 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8058101ba0 0x7f80581124d0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f804c003080 tx=0x7f804c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:36:28.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:28.079+0000 7f80557fa640 1 -- 172.21.15.1:0/256882161 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f804c038070 con 0x7f8058101ba0 2024-06-18T02:36:28.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:28.079+0000 7f805eb19640 1 -- 172.21.15.1:0/256882161 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f805810c190 con 0x7f8058101ba0 2024-06-18T02:36:28.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:28.079+0000 7f80557fa640 1 -- 172.21.15.1:0/256882161 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f804c008ee0 con 0x7f8058101ba0 2024-06-18T02:36:28.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:28.079+0000 7f80557fa640 1 -- 172.21.15.1:0/256882161 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f804c03d870 con 0x7f8058101ba0 2024-06-18T02:36:28.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:28.079+0000 7f805eb19640 1 -- 172.21.15.1:0/256882161 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f805810c650 con 0x7f8058101ba0 2024-06-18T02:36:28.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:28.080+0000 7f805eb19640 1 -- 172.21.15.1:0/256882161 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80580fc970 con 0x7f8058101ba0 2024-06-18T02:36:28.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:28.081+0000 7f80557fa640 1 -- 172.21.15.1:0/256882161 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7f804c030020 con 0x7f8058101ba0 2024-06-18T02:36:28.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:28.081+0000 7f80557fa640 1 --2- 172.21.15.1:0/256882161 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f8040065430 0x7f80400678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:36:28.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:28.081+0000 7f80557fa640 1 -- 172.21.15.1:0/256882161 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f804c0a2070 con 0x7f8058101ba0 2024-06-18T02:36:28.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:28.081+0000 7f805c8b5640 1 --2- 172.21.15.1:0/256882161 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f8040065430 0x7f80400678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:36:28.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:28.082+0000 7f805c8b5640 1 --2- 172.21.15.1:0/256882161 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f8040065430 0x7f80400678e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8048009ac0 tx=0x7f8048009120 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:36:28.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:28.086+0000 7f80557fa640 1 -- 172.21.15.1:0/256882161 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f804c070640 con 0x7f8058101ba0 2024-06-18T02:36:28.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:36:27 smithi028 ceph-mon[35823]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:36:28.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:28.298+0000 7f805eb19640 1 -- 172.21.15.1:0/256882161 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f80581083f0 con 0x7f8058101ba0 2024-06-18T02:36:28.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:28.298+0000 7f80557fa640 1 -- 172.21.15.1:0/256882161 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f804c073c60 con 0x7f8058101ba0 2024-06-18T02:36:28.298 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:36:28.298 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-18T02:36:28.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:28.300+0000 7f805eb19640 1 -- 172.21.15.1:0/256882161 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f8040065430 msgr2=0x7f80400678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:36:28.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:28.300+0000 7f805eb19640 1 --2- 172.21.15.1:0/256882161 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f8040065430 0x7f80400678e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8048009ac0 tx=0x7f8048009120 comp rx=0 tx=0).stop 2024-06-18T02:36:28.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:28.300+0000 7f805eb19640 1 -- 172.21.15.1:0/256882161 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8058101ba0 msgr2=0x7f80581124d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:36:28.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:28.300+0000 7f805eb19640 1 --2- 172.21.15.1:0/256882161 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8058101ba0 0x7f80581124d0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f804c003080 tx=0x7f804c009da0 comp rx=0 tx=0).stop 2024-06-18T02:36:28.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:28.300+0000 7f805eb19640 1 -- 172.21.15.1:0/256882161 shutdown_connections 2024-06-18T02:36:28.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:28.300+0000 7f805eb19640 1 --2- 172.21.15.1:0/256882161 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f8040065430 0x7f80400678e0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:28.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:28.300+0000 7f805eb19640 1 --2- 172.21.15.1:0/256882161 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8058101ba0 0x7f80581124d0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:28.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:28.300+0000 7f805eb19640 1 --2- 172.21.15.1:0/256882161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8058101200 0x7f8058111f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:28.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:28.301+0000 7f805eb19640 1 -- 172.21.15.1:0/256882161 >> 172.21.15.1:0/256882161 conn(0x7f80580f6df0 msgr2=0x7f80580f7db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:36:28.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:28.301+0000 7f805eb19640 1 -- 172.21.15.1:0/256882161 shutdown_connections 2024-06-18T02:36:28.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:28.301+0000 7f805eb19640 1 -- 172.21.15.1:0/256882161 wait complete. 2024-06-18T02:36:29.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:36:28 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/256882161' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:36:29.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:36:28 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/256882161' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:36:30.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:36:29 smithi028 ceph-mon[35823]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:36:30.377 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:36:29 smithi001 ceph-mon[29099]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:36:31.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:36:30 smithi028 ceph-mon[35823]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:36:31.377 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:36:30 smithi001 ceph-mon[29099]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:36:31.964 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T02:36:32.240 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:36:33.715 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:36:33 smithi001 ceph-mon[29099]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:36:33.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:33.941+0000 7f83308a3640 1 -- 172.21.15.1:0/3744847499 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f832c1022f0 msgr2=0x7f832c1026c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:36:33.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:33.941+0000 7f83308a3640 1 --2- 172.21.15.1:0/3744847499 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f832c1022f0 0x7f832c1026c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f831c0048d0 tx=0x7f831c009da0 comp rx=0 tx=0).stop 2024-06-18T02:36:33.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:33.941+0000 7f83308a3640 1 -- 172.21.15.1:0/3744847499 shutdown_connections 2024-06-18T02:36:33.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:33.941+0000 7f83308a3640 1 --2- 172.21.15.1:0/3744847499 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f832c102c90 0x7f832c0fa9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:33.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:33.941+0000 7f83308a3640 1 --2- 172.21.15.1:0/3744847499 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f832c1022f0 0x7f832c1026c0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:33.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:33.941+0000 7f83308a3640 1 -- 172.21.15.1:0/3744847499 >> 172.21.15.1:0/3744847499 conn(0x7f832c070a40 msgr2=0x7f832c070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:36:33.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:33.941+0000 7f83308a3640 1 -- 172.21.15.1:0/3744847499 shutdown_connections 2024-06-18T02:36:33.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:33.941+0000 7f83308a3640 1 -- 172.21.15.1:0/3744847499 wait complete. 2024-06-18T02:36:33.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:33.942+0000 7f83308a3640 1 Processor -- start 2024-06-18T02:36:33.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:33.942+0000 7f83308a3640 1 -- start start 2024-06-18T02:36:33.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:33.942+0000 7f83308a3640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f832c1022f0 0x7f832c113170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:36:33.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:33.942+0000 7f83308a3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f832c102c90 0x7f832c1136b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:36:33.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:33.942+0000 7f83308a3640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f832c113d40 con 0x7f832c1022f0 2024-06-18T02:36:33.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:33.942+0000 7f83308a3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f832c10d0f0 con 0x7f832c102c90 2024-06-18T02:36:33.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:33.942+0000 7f832a59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f832c1022f0 0x7f832c113170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:36:33.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:33.942+0000 7f832a59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f832c1022f0 0x7f832c113170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51978/0 (socket says 172.21.15.1:51978) 2024-06-18T02:36:33.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:33.942+0000 7f832a59c640 1 -- 172.21.15.1:0/1019377984 learned_addr learned my addr 172.21.15.1:0/1019377984 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:36:33.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:33.942+0000 7f8329d9b640 1 --2- 172.21.15.1:0/1019377984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f832c102c90 0x7f832c1136b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:36:33.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:33.942+0000 7f832a59c640 1 -- 172.21.15.1:0/1019377984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f832c102c90 msgr2=0x7f832c1136b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:36:33.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:33.942+0000 7f832a59c640 1 --2- 172.21.15.1:0/1019377984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f832c102c90 0x7f832c1136b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:33.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:33.942+0000 7f832a59c640 1 -- 172.21.15.1:0/1019377984 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f831c005040 con 0x7f832c1022f0 2024-06-18T02:36:33.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:33.942+0000 7f832a59c640 1 --2- 172.21.15.1:0/1019377984 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f832c1022f0 0x7f832c113170 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f831c008fc0 tx=0x7f831c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:36:33.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:33.943+0000 7f831affd640 1 -- 172.21.15.1:0/1019377984 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f831c038070 con 0x7f832c1022f0 2024-06-18T02:36:33.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:33.943+0000 7f83308a3640 1 -- 172.21.15.1:0/1019377984 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f832c10d370 con 0x7f832c1022f0 2024-06-18T02:36:33.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:33.943+0000 7f831affd640 1 -- 172.21.15.1:0/1019377984 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f831c032b80 con 0x7f832c1022f0 2024-06-18T02:36:33.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:33.943+0000 7f831affd640 1 -- 172.21.15.1:0/1019377984 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f831c03dc20 con 0x7f832c1022f0 2024-06-18T02:36:33.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:33.943+0000 7f83308a3640 1 -- 172.21.15.1:0/1019377984 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f832c10d830 con 0x7f832c1022f0 2024-06-18T02:36:33.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:33.944+0000 7f83308a3640 1 -- 172.21.15.1:0/1019377984 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f832c065510 con 0x7f832c1022f0 2024-06-18T02:36:33.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:33.944+0000 7f831affd640 1 -- 172.21.15.1:0/1019377984 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7f831c030020 con 0x7f832c1022f0 2024-06-18T02:36:33.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:33.944+0000 7f831affd640 1 --2- 172.21.15.1:0/1019377984 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f8314065430 0x7f83140678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:36:33.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:33.945+0000 7f8329d9b640 1 --2- 172.21.15.1:0/1019377984 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f8314065430 0x7f83140678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:36:33.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:33.945+0000 7f831affd640 1 -- 172.21.15.1:0/1019377984 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f831c0a2f70 con 0x7f832c1022f0 2024-06-18T02:36:33.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:33.945+0000 7f8329d9b640 1 --2- 172.21.15.1:0/1019377984 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f8314065430 0x7f83140678e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f832c10ea30 tx=0x7f8320009620 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:36:33.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:33.949+0000 7f831affd640 1 -- 172.21.15.1:0/1019377984 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f831c071380 con 0x7f832c1022f0 2024-06-18T02:36:33.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:36:33 smithi028 ceph-mon[35823]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:36:34.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:34.159+0000 7f83308a3640 1 -- 172.21.15.1:0/1019377984 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f832c04b510 con 0x7f832c1022f0 2024-06-18T02:36:34.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:34.159+0000 7f831affd640 1 -- 172.21.15.1:0/1019377984 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f831c0749a0 con 0x7f832c1022f0 2024-06-18T02:36:34.159 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:36:34.159 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-18T02:36:34.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:34.161+0000 7f83308a3640 1 -- 172.21.15.1:0/1019377984 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f8314065430 msgr2=0x7f83140678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:36:34.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:34.161+0000 7f83308a3640 1 --2- 172.21.15.1:0/1019377984 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f8314065430 0x7f83140678e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f832c10ea30 tx=0x7f8320009620 comp rx=0 tx=0).stop 2024-06-18T02:36:34.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:34.161+0000 7f83308a3640 1 -- 172.21.15.1:0/1019377984 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f832c1022f0 msgr2=0x7f832c113170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:36:34.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:34.161+0000 7f83308a3640 1 --2- 172.21.15.1:0/1019377984 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f832c1022f0 0x7f832c113170 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f831c008fc0 tx=0x7f831c009da0 comp rx=0 tx=0).stop 2024-06-18T02:36:34.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:34.161+0000 7f83308a3640 1 -- 172.21.15.1:0/1019377984 shutdown_connections 2024-06-18T02:36:34.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:34.161+0000 7f83308a3640 1 --2- 172.21.15.1:0/1019377984 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f8314065430 0x7f83140678e0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:34.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:34.161+0000 7f83308a3640 1 --2- 172.21.15.1:0/1019377984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f832c102c90 0x7f832c1136b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:34.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:34.161+0000 7f83308a3640 1 --2- 172.21.15.1:0/1019377984 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f832c1022f0 0x7f832c113170 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:34.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:34.161+0000 7f83308a3640 1 -- 172.21.15.1:0/1019377984 >> 172.21.15.1:0/1019377984 conn(0x7f832c070a40 msgr2=0x7f832c1050b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:36:34.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:34.162+0000 7f83308a3640 1 -- 172.21.15.1:0/1019377984 shutdown_connections 2024-06-18T02:36:34.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:34.162+0000 7f83308a3640 1 -- 172.21.15.1:0/1019377984 wait complete. 2024-06-18T02:36:34.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:36:34 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/1019377984' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:36:34.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:36:34 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/1019377984' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:36:35.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:36:35 smithi001 ceph-mon[29099]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:36:35.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:36:35 smithi028 ceph-mon[35823]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:36:36.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:36:36 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:36:37.127 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:36:36 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:36:37.765 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T02:36:37.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:36:37 smithi028 ceph-mon[35823]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:36:38.041 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:36:38.127 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:36:37 smithi001 ceph-mon[29099]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:36:39.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:39.720+0000 7f4306f9c640 1 -- 172.21.15.1:0/1559142033 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4300101b80 msgr2=0x7f43000fb210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:36:39.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:39.720+0000 7f4306f9c640 1 --2- 172.21.15.1:0/1559142033 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4300101b80 0x7f43000fb210 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f42ec0048d0 tx=0x7f42ec009da0 comp rx=0 tx=0).stop 2024-06-18T02:36:39.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:39.720+0000 7f4306f9c640 1 -- 172.21.15.1:0/1559142033 shutdown_connections 2024-06-18T02:36:39.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:39.720+0000 7f4306f9c640 1 --2- 172.21.15.1:0/1559142033 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4300101b80 0x7f43000fb210 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:39.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:39.720+0000 7f4306f9c640 1 --2- 172.21.15.1:0/1559142033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43001011e0 0x7f43001015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:39.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:39.720+0000 7f4306f9c640 1 -- 172.21.15.1:0/1559142033 >> 172.21.15.1:0/1559142033 conn(0x7f43000f6df0 msgr2=0x7f43000f9200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:36:39.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:39.720+0000 7f4306f9c640 1 -- 172.21.15.1:0/1559142033 shutdown_connections 2024-06-18T02:36:39.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:39.720+0000 7f4306f9c640 1 -- 172.21.15.1:0/1559142033 wait complete. 2024-06-18T02:36:39.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:39.721+0000 7f4306f9c640 1 Processor -- start 2024-06-18T02:36:39.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:39.721+0000 7f4306f9c640 1 -- start start 2024-06-18T02:36:39.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:39.721+0000 7f4306f9c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43001011e0 0x7f4300113060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:36:39.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:39.721+0000 7f4306f9c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4300101b80 0x7f43001135a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:36:39.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:39.721+0000 7f4306f9c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4300113ba0 con 0x7f4300101b80 2024-06-18T02:36:39.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:39.721+0000 7f4306f9c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f430010cfe0 con 0x7f43001011e0 2024-06-18T02:36:39.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:39.721+0000 7f42fffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4300101b80 0x7f43001135a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:36:39.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:39.721+0000 7f42fffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4300101b80 0x7f43001135a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52000/0 (socket says 172.21.15.1:52000) 2024-06-18T02:36:39.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:39.721+0000 7f42fffff640 1 -- 172.21.15.1:0/4135985140 learned_addr learned my addr 172.21.15.1:0/4135985140 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:36:39.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:39.721+0000 7f4304d38640 1 --2- 172.21.15.1:0/4135985140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43001011e0 0x7f4300113060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:36:39.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:39.722+0000 7f42fffff640 1 -- 172.21.15.1:0/4135985140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43001011e0 msgr2=0x7f4300113060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:36:39.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:39.722+0000 7f42fffff640 1 --2- 172.21.15.1:0/4135985140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43001011e0 0x7f4300113060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:39.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:39.722+0000 7f42fffff640 1 -- 172.21.15.1:0/4135985140 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42ec005040 con 0x7f4300101b80 2024-06-18T02:36:39.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:39.722+0000 7f42fffff640 1 --2- 172.21.15.1:0/4135985140 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4300101b80 0x7f43001135a0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f42ec0043d0 tx=0x7f42ec009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:36:39.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:39.722+0000 7f42fd7fa640 1 -- 172.21.15.1:0/4135985140 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f42ec038070 con 0x7f4300101b80 2024-06-18T02:36:39.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:39.722+0000 7f4306f9c640 1 -- 172.21.15.1:0/4135985140 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f430010d260 con 0x7f4300101b80 2024-06-18T02:36:39.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:39.722+0000 7f42fd7fa640 1 -- 172.21.15.1:0/4135985140 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f42ec032970 con 0x7f4300101b80 2024-06-18T02:36:39.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:39.722+0000 7f42fd7fa640 1 -- 172.21.15.1:0/4135985140 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f42ec03da60 con 0x7f4300101b80 2024-06-18T02:36:39.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:39.722+0000 7f4306f9c640 1 -- 172.21.15.1:0/4135985140 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f430010d7a0 con 0x7f4300101b80 2024-06-18T02:36:39.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:39.723+0000 7f4306f9c640 1 -- 172.21.15.1:0/4135985140 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43000fc950 con 0x7f4300101b80 2024-06-18T02:36:39.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:39.724+0000 7f42fd7fa640 1 -- 172.21.15.1:0/4135985140 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7f42ec030020 con 0x7f4300101b80 2024-06-18T02:36:39.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:39.724+0000 7f42fd7fa640 1 --2- 172.21.15.1:0/4135985140 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f42f0065430 0x7f42f00678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:36:39.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:39.725+0000 7f42fd7fa640 1 -- 172.21.15.1:0/4135985140 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f42ec0a25b0 con 0x7f4300101b80 2024-06-18T02:36:39.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:39.725+0000 7f4304d38640 1 --2- 172.21.15.1:0/4135985140 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f42f0065430 0x7f42f00678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:36:39.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:39.725+0000 7f4304d38640 1 --2- 172.21.15.1:0/4135985140 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f42f0065430 0x7f42f00678e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f42f4004ed0 tx=0x7f42f400b040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:36:39.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:39.729+0000 7f42fd7fa640 1 -- 172.21.15.1:0/4135985140 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f42ec070a60 con 0x7f4300101b80 2024-06-18T02:36:39.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:36:39 smithi001 ceph-mon[29099]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:36:39.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:39.940+0000 7f4306f9c640 1 -- 172.21.15.1:0/4135985140 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4300109440 con 0x7f4300101b80 2024-06-18T02:36:39.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:39.940+0000 7f42fd7fa640 1 -- 172.21.15.1:0/4135985140 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f42ec074080 con 0x7f4300101b80 2024-06-18T02:36:39.940 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:36:39.940 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-18T02:36:39.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:39.942+0000 7f4306f9c640 1 -- 172.21.15.1:0/4135985140 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f42f0065430 msgr2=0x7f42f00678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:36:39.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:39.942+0000 7f4306f9c640 1 --2- 172.21.15.1:0/4135985140 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f42f0065430 0x7f42f00678e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f42f4004ed0 tx=0x7f42f400b040 comp rx=0 tx=0).stop 2024-06-18T02:36:39.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:39.942+0000 7f4306f9c640 1 -- 172.21.15.1:0/4135985140 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4300101b80 msgr2=0x7f43001135a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:36:39.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:39.942+0000 7f4306f9c640 1 --2- 172.21.15.1:0/4135985140 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4300101b80 0x7f43001135a0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f42ec0043d0 tx=0x7f42ec009da0 comp rx=0 tx=0).stop 2024-06-18T02:36:39.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:39.942+0000 7f4306f9c640 1 -- 172.21.15.1:0/4135985140 shutdown_connections 2024-06-18T02:36:39.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:39.942+0000 7f4306f9c640 1 --2- 172.21.15.1:0/4135985140 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f42f0065430 0x7f42f00678e0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:39.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:39.942+0000 7f4306f9c640 1 --2- 172.21.15.1:0/4135985140 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4300101b80 0x7f43001135a0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:39.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:39.942+0000 7f4306f9c640 1 --2- 172.21.15.1:0/4135985140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43001011e0 0x7f4300113060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:39.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:39.942+0000 7f4306f9c640 1 -- 172.21.15.1:0/4135985140 >> 172.21.15.1:0/4135985140 conn(0x7f43000f6df0 msgr2=0x7f43000f7d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:36:39.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:39.942+0000 7f4306f9c640 1 -- 172.21.15.1:0/4135985140 shutdown_connections 2024-06-18T02:36:39.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:39.942+0000 7f4306f9c640 1 -- 172.21.15.1:0/4135985140 wait complete. 2024-06-18T02:36:39.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:36:39 smithi028 ceph-mon[35823]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:36:40.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:36:40 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/4135985140' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:36:40.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:36:40 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/4135985140' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:36:41.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:36:41 smithi028 ceph-mon[35823]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:36:42.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:36:41 smithi001 ceph-mon[29099]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:36:43.587 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T02:36:43.864 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:36:43.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:36:43 smithi001 ceph-mon[29099]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:36:43.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:36:43 smithi028 ceph-mon[35823]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:36:45.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:45.559+0000 7fda34e11640 1 -- 172.21.15.1:0/2412217048 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda301022c0 msgr2=0x7fda30102690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:36:45.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:45.559+0000 7fda34e11640 1 --2- 172.21.15.1:0/2412217048 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda301022c0 0x7fda30102690 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fda200048d0 tx=0x7fda20009da0 comp rx=0 tx=0).stop 2024-06-18T02:36:45.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:45.559+0000 7fda34e11640 1 -- 172.21.15.1:0/2412217048 shutdown_connections 2024-06-18T02:36:45.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:45.559+0000 7fda34e11640 1 --2- 172.21.15.1:0/2412217048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda30102c60 0x7fda300fc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:45.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:45.559+0000 7fda34e11640 1 --2- 172.21.15.1:0/2412217048 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda301022c0 0x7fda30102690 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:45.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:45.559+0000 7fda34e11640 1 -- 172.21.15.1:0/2412217048 >> 172.21.15.1:0/2412217048 conn(0x7fda300f7e50 msgr2=0x7fda300fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:36:45.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:45.560+0000 7fda34e11640 1 -- 172.21.15.1:0/2412217048 shutdown_connections 2024-06-18T02:36:45.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:45.560+0000 7fda34e11640 1 -- 172.21.15.1:0/2412217048 wait complete. 2024-06-18T02:36:45.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:45.560+0000 7fda34e11640 1 Processor -- start 2024-06-18T02:36:45.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:45.560+0000 7fda34e11640 1 -- start start 2024-06-18T02:36:45.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:45.560+0000 7fda34e11640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda301022c0 0x7fda301133d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:36:45.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:45.561+0000 7fda34e11640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda30102c60 0x7fda30113910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:36:45.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:45.561+0000 7fda34e11640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda30113fa0 con 0x7fda301022c0 2024-06-18T02:36:45.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:45.561+0000 7fda34e11640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda3010d350 con 0x7fda30102c60 2024-06-18T02:36:45.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:45.561+0000 7fda2e59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda301022c0 0x7fda301133d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:36:45.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:45.561+0000 7fda2e59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda301022c0 0x7fda301133d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47898/0 (socket says 172.21.15.1:47898) 2024-06-18T02:36:45.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:45.561+0000 7fda2e59c640 1 -- 172.21.15.1:0/3727968571 learned_addr learned my addr 172.21.15.1:0/3727968571 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:36:45.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:45.561+0000 7fda2e59c640 1 -- 172.21.15.1:0/3727968571 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda30102c60 msgr2=0x7fda30113910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:36:45.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:45.561+0000 7fda2dd9b640 1 --2- 172.21.15.1:0/3727968571 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda30102c60 0x7fda30113910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:36:45.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:45.561+0000 7fda2e59c640 1 --2- 172.21.15.1:0/3727968571 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda30102c60 0x7fda30113910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:45.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:45.561+0000 7fda2e59c640 1 -- 172.21.15.1:0/3727968571 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda20005040 con 0x7fda301022c0 2024-06-18T02:36:45.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:45.561+0000 7fda2e59c640 1 --2- 172.21.15.1:0/3727968571 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda301022c0 0x7fda301133d0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fda20003080 tx=0x7fda20009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:36:45.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:45.561+0000 7fda1effd640 1 -- 172.21.15.1:0/3727968571 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fda20038070 con 0x7fda301022c0 2024-06-18T02:36:45.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:45.561+0000 7fda1effd640 1 -- 172.21.15.1:0/3727968571 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fda20008ee0 con 0x7fda301022c0 2024-06-18T02:36:45.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:45.561+0000 7fda34e11640 1 -- 172.21.15.1:0/3727968571 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda3010d5d0 con 0x7fda301022c0 2024-06-18T02:36:45.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:45.561+0000 7fda1effd640 1 -- 172.21.15.1:0/3727968571 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fda2003d870 con 0x7fda301022c0 2024-06-18T02:36:45.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:45.561+0000 7fda34e11640 1 -- 172.21.15.1:0/3727968571 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda3010da90 con 0x7fda301022c0 2024-06-18T02:36:45.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:45.563+0000 7fda34e11640 1 -- 172.21.15.1:0/3727968571 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda300fda30 con 0x7fda301022c0 2024-06-18T02:36:45.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:45.563+0000 7fda1effd640 1 -- 172.21.15.1:0/3727968571 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7fda20030020 con 0x7fda301022c0 2024-06-18T02:36:45.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:45.563+0000 7fda1effd640 1 --2- 172.21.15.1:0/3727968571 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fda18065430 0x7fda180678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:36:45.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:45.563+0000 7fda1effd640 1 -- 172.21.15.1:0/3727968571 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fda200a2230 con 0x7fda301022c0 2024-06-18T02:36:45.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:45.563+0000 7fda2dd9b640 1 --2- 172.21.15.1:0/3727968571 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fda18065430 0x7fda180678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:36:45.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:45.564+0000 7fda2dd9b640 1 --2- 172.21.15.1:0/3727968571 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fda18065430 0x7fda180678e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fda24006fd0 tx=0x7fda24008040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:36:45.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:45.568+0000 7fda1effd640 1 -- 172.21.15.1:0/3727968571 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fda20070760 con 0x7fda301022c0 2024-06-18T02:36:45.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:45.776+0000 7fda34e11640 1 -- 172.21.15.1:0/3727968571 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fda3004b510 con 0x7fda301022c0 2024-06-18T02:36:45.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:45.776+0000 7fda1effd640 1 -- 172.21.15.1:0/3727968571 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fda20073d80 con 0x7fda301022c0 2024-06-18T02:36:45.776 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:36:45.776 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-18T02:36:45.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:45.778+0000 7fda34e11640 1 -- 172.21.15.1:0/3727968571 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fda18065430 msgr2=0x7fda180678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:36:45.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:45.778+0000 7fda34e11640 1 --2- 172.21.15.1:0/3727968571 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fda18065430 0x7fda180678e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fda24006fd0 tx=0x7fda24008040 comp rx=0 tx=0).stop 2024-06-18T02:36:45.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:45.778+0000 7fda34e11640 1 -- 172.21.15.1:0/3727968571 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda301022c0 msgr2=0x7fda301133d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:36:45.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:45.778+0000 7fda34e11640 1 --2- 172.21.15.1:0/3727968571 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda301022c0 0x7fda301133d0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fda20003080 tx=0x7fda20009da0 comp rx=0 tx=0).stop 2024-06-18T02:36:45.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:45.778+0000 7fda34e11640 1 -- 172.21.15.1:0/3727968571 shutdown_connections 2024-06-18T02:36:45.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:45.778+0000 7fda34e11640 1 --2- 172.21.15.1:0/3727968571 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fda18065430 0x7fda180678e0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:45.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:45.778+0000 7fda34e11640 1 --2- 172.21.15.1:0/3727968571 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda30102c60 0x7fda30113910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:45.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:45.778+0000 7fda34e11640 1 --2- 172.21.15.1:0/3727968571 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fda301022c0 0x7fda301133d0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:45.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:45.778+0000 7fda34e11640 1 -- 172.21.15.1:0/3727968571 >> 172.21.15.1:0/3727968571 conn(0x7fda300f7e50 msgr2=0x7fda300f8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:36:45.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:45.778+0000 7fda34e11640 1 -- 172.21.15.1:0/3727968571 shutdown_connections 2024-06-18T02:36:45.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:45.778+0000 7fda34e11640 1 -- 172.21.15.1:0/3727968571 wait complete. 2024-06-18T02:36:45.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:36:45 smithi028 ceph-mon[35823]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:36:46.090 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:36:45 smithi001 ceph-mon[29099]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:36:46.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:36:46 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/3727968571' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:36:47.127 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:36:46 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/3727968571' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:36:47.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:36:47 smithi028 ceph-mon[35823]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:36:48.127 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:36:47 smithi001 ceph-mon[29099]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:36:49.375 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T02:36:49.653 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:36:49.907 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:36:49 smithi001 ceph-mon[29099]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:36:49.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:36:49 smithi028 ceph-mon[35823]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:36:51.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:51.238+0000 7f405f9c9640 1 -- 172.21.15.1:0/1705882613 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4058102c00 msgr2=0x7f40580fc290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:36:51.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:51.238+0000 7f405f9c9640 1 --2- 172.21.15.1:0/1705882613 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4058102c00 0x7f40580fc290 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f40480048d0 tx=0x7f4048009da0 comp rx=0 tx=0).stop 2024-06-18T02:36:51.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:51.238+0000 7f405f9c9640 1 -- 172.21.15.1:0/1705882613 shutdown_connections 2024-06-18T02:36:51.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:51.238+0000 7f405f9c9640 1 --2- 172.21.15.1:0/1705882613 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4058102c00 0x7f40580fc290 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:51.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:51.238+0000 7f405f9c9640 1 --2- 172.21.15.1:0/1705882613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4058102260 0x7f4058102630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:51.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:51.238+0000 7f405f9c9640 1 -- 172.21.15.1:0/1705882613 >> 172.21.15.1:0/1705882613 conn(0x7f40580f7e50 msgr2=0x7f40580fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:36:51.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:51.238+0000 7f405f9c9640 1 -- 172.21.15.1:0/1705882613 shutdown_connections 2024-06-18T02:36:51.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:51.238+0000 7f405f9c9640 1 -- 172.21.15.1:0/1705882613 wait complete. 2024-06-18T02:36:51.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:51.239+0000 7f405f9c9640 1 Processor -- start 2024-06-18T02:36:51.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:51.239+0000 7f405f9c9640 1 -- start start 2024-06-18T02:36:51.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:51.239+0000 7f405f9c9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4058102260 0x7f40581151a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:36:51.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:51.239+0000 7f405f9c9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4058102c00 0x7f40581156e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:36:51.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:51.239+0000 7f405f9c9640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4058115ce0 con 0x7f4058102c00 2024-06-18T02:36:51.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:51.239+0000 7f405f9c9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f405810f120 con 0x7f4058102260 2024-06-18T02:36:51.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:51.239+0000 7f405cf64640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4058102c00 0x7f40581156e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:36:51.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:51.239+0000 7f405cf64640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4058102c00 0x7f40581156e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47912/0 (socket says 172.21.15.1:47912) 2024-06-18T02:36:51.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:51.239+0000 7f405cf64640 1 -- 172.21.15.1:0/1861260494 learned_addr learned my addr 172.21.15.1:0/1861260494 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:36:51.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:51.239+0000 7f405d765640 1 --2- 172.21.15.1:0/1861260494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4058102260 0x7f40581151a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:36:51.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:51.240+0000 7f405cf64640 1 -- 172.21.15.1:0/1861260494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4058102260 msgr2=0x7f40581151a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:36:51.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:51.240+0000 7f405cf64640 1 --2- 172.21.15.1:0/1861260494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4058102260 0x7f40581151a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:51.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:51.240+0000 7f405cf64640 1 -- 172.21.15.1:0/1861260494 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4048005040 con 0x7f4058102c00 2024-06-18T02:36:51.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:51.240+0000 7f405d765640 1 --2- 172.21.15.1:0/1861260494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4058102260 0x7f40581151a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-18T02:36:51.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:51.240+0000 7f405cf64640 1 --2- 172.21.15.1:0/1861260494 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4058102c00 0x7f40581156e0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f40480043d0 tx=0x7f4048009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:36:51.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:51.240+0000 7f404dffb640 1 -- 172.21.15.1:0/1861260494 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4048038070 con 0x7f4058102c00 2024-06-18T02:36:51.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:51.240+0000 7f404dffb640 1 -- 172.21.15.1:0/1861260494 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4048032970 con 0x7f4058102c00 2024-06-18T02:36:51.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:51.240+0000 7f405f9c9640 1 -- 172.21.15.1:0/1861260494 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f405810f3a0 con 0x7f4058102c00 2024-06-18T02:36:51.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:51.240+0000 7f404dffb640 1 -- 172.21.15.1:0/1861260494 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f404803da60 con 0x7f4058102c00 2024-06-18T02:36:51.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:51.240+0000 7f405f9c9640 1 -- 172.21.15.1:0/1861260494 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f405810f8e0 con 0x7f4058102c00 2024-06-18T02:36:51.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:51.241+0000 7f405f9c9640 1 -- 172.21.15.1:0/1861260494 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40580fd9d0 con 0x7f4058102c00 2024-06-18T02:36:51.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:51.241+0000 7f404dffb640 1 -- 172.21.15.1:0/1861260494 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7f4048030020 con 0x7f4058102c00 2024-06-18T02:36:51.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:51.242+0000 7f404dffb640 1 --2- 172.21.15.1:0/1861260494 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f4044065430 0x7f40440678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:36:51.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:51.242+0000 7f404dffb640 1 -- 172.21.15.1:0/1861260494 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f40480a2590 con 0x7f4058102c00 2024-06-18T02:36:51.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:51.242+0000 7f405d765640 1 --2- 172.21.15.1:0/1861260494 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f4044065430 0x7f40440678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:36:51.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:51.242+0000 7f405d765640 1 --2- 172.21.15.1:0/1861260494 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f4044065430 0x7f40440678e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4054007fd0 tx=0x7f4054009e90 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:36:51.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:51.246+0000 7f404dffb640 1 -- 172.21.15.1:0/1861260494 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f4048070a40 con 0x7f4058102c00 2024-06-18T02:36:51.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:51.455+0000 7f405f9c9640 1 -- 172.21.15.1:0/1861260494 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f40581093c0 con 0x7f4058102c00 2024-06-18T02:36:51.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:51.455+0000 7f404dffb640 1 -- 172.21.15.1:0/1861260494 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f4048074060 con 0x7f4058102c00 2024-06-18T02:36:51.455 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:36:51.455 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-18T02:36:51.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:51.457+0000 7f405f9c9640 1 -- 172.21.15.1:0/1861260494 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f4044065430 msgr2=0x7f40440678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:36:51.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:51.457+0000 7f405f9c9640 1 --2- 172.21.15.1:0/1861260494 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f4044065430 0x7f40440678e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4054007fd0 tx=0x7f4054009e90 comp rx=0 tx=0).stop 2024-06-18T02:36:51.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:51.457+0000 7f405f9c9640 1 -- 172.21.15.1:0/1861260494 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4058102c00 msgr2=0x7f40581156e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:36:51.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:51.457+0000 7f405f9c9640 1 --2- 172.21.15.1:0/1861260494 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4058102c00 0x7f40581156e0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f40480043d0 tx=0x7f4048009da0 comp rx=0 tx=0).stop 2024-06-18T02:36:51.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:51.457+0000 7f405f9c9640 1 -- 172.21.15.1:0/1861260494 shutdown_connections 2024-06-18T02:36:51.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:51.457+0000 7f405f9c9640 1 --2- 172.21.15.1:0/1861260494 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f4044065430 0x7f40440678e0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:51.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:51.457+0000 7f405f9c9640 1 --2- 172.21.15.1:0/1861260494 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4058102c00 0x7f40581156e0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:51.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:51.457+0000 7f405f9c9640 1 --2- 172.21.15.1:0/1861260494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4058102260 0x7f40581151a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:51.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:51.457+0000 7f405f9c9640 1 -- 172.21.15.1:0/1861260494 >> 172.21.15.1:0/1861260494 conn(0x7f40580f7e50 msgr2=0x7f4058071140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:36:51.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:51.457+0000 7f405f9c9640 1 -- 172.21.15.1:0/1861260494 shutdown_connections 2024-06-18T02:36:51.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:51.457+0000 7f405f9c9640 1 -- 172.21.15.1:0/1861260494 wait complete. 2024-06-18T02:36:51.818 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:36:51 smithi001 ceph-mon[29099]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:36:51.819 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:36:51 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/1861260494' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:36:51.819 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:36:51 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:36:51.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:36:51 smithi028 ceph-mon[35823]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:36:51.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:36:51 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/1861260494' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:36:51.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:36:51 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:36:53.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:36:53 smithi001 ceph-mon[29099]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:36:53.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:36:53 smithi028 ceph-mon[35823]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:36:55.146 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T02:36:55.423 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:36:55.871 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:36:55 smithi001 ceph-mon[29099]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:36:55.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:36:55 smithi028 ceph-mon[35823]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:36:57.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:57.091+0000 7f9c4cecc640 1 -- 172.21.15.1:0/3337227982 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c48070120 msgr2=0x7f9c481085f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:36:57.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:57.091+0000 7f9c4cecc640 1 --2- 172.21.15.1:0/3337227982 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c48070120 0x7f9c481085f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9c300048d0 tx=0x7f9c30009da0 comp rx=0 tx=0).stop 2024-06-18T02:36:57.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:57.091+0000 7f9c4cecc640 1 -- 172.21.15.1:0/3337227982 shutdown_connections 2024-06-18T02:36:57.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:57.091+0000 7f9c4cecc640 1 --2- 172.21.15.1:0/3337227982 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c48070120 0x7f9c481085f0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:57.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:57.091+0000 7f9c4cecc640 1 --2- 172.21.15.1:0/3337227982 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c4806f810 0x7f9c4806fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:57.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:57.091+0000 7f9c4cecc640 1 -- 172.21.15.1:0/3337227982 >> 172.21.15.1:0/3337227982 conn(0x7f9c480f7e50 msgr2=0x7f9c480fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:36:57.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:57.091+0000 7f9c4cecc640 1 -- 172.21.15.1:0/3337227982 shutdown_connections 2024-06-18T02:36:57.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:57.091+0000 7f9c4cecc640 1 -- 172.21.15.1:0/3337227982 wait complete. 2024-06-18T02:36:57.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:57.092+0000 7f9c4cecc640 1 Processor -- start 2024-06-18T02:36:57.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:57.092+0000 7f9c4cecc640 1 -- start start 2024-06-18T02:36:57.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:57.092+0000 7f9c4cecc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c4806f810 0x7f9c481155e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:36:57.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:57.092+0000 7f9c4cecc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c48070120 0x7f9c48115b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:36:57.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:57.092+0000 7f9c4cecc640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c48116220 con 0x7f9c4806f810 2024-06-18T02:36:57.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:57.092+0000 7f9c4cecc640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c4810f580 con 0x7f9c48070120 2024-06-18T02:36:57.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:57.092+0000 7f9c4659c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c4806f810 0x7f9c481155e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:36:57.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:57.092+0000 7f9c4659c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c4806f810 0x7f9c481155e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53712/0 (socket says 172.21.15.1:53712) 2024-06-18T02:36:57.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:57.092+0000 7f9c4659c640 1 -- 172.21.15.1:0/552023604 learned_addr learned my addr 172.21.15.1:0/552023604 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:36:57.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:57.092+0000 7f9c4659c640 1 -- 172.21.15.1:0/552023604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c48070120 msgr2=0x7f9c48115b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:36:57.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:57.092+0000 7f9c4659c640 1 --2- 172.21.15.1:0/552023604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c48070120 0x7f9c48115b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:57.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:57.092+0000 7f9c4659c640 1 -- 172.21.15.1:0/552023604 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c30005040 con 0x7f9c4806f810 2024-06-18T02:36:57.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:57.093+0000 7f9c4659c640 1 --2- 172.21.15.1:0/552023604 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c4806f810 0x7f9c481155e0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f9c3800b790 tx=0x7f9c380091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:36:57.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:57.093+0000 7f9c3effd640 1 -- 172.21.15.1:0/552023604 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c3800c700 con 0x7f9c4806f810 2024-06-18T02:36:57.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:57.093+0000 7f9c4cecc640 1 -- 172.21.15.1:0/552023604 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c4810f860 con 0x7f9c4806f810 2024-06-18T02:36:57.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:57.093+0000 7f9c3effd640 1 -- 172.21.15.1:0/552023604 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9c3800cd30 con 0x7f9c4806f810 2024-06-18T02:36:57.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:57.093+0000 7f9c3effd640 1 -- 172.21.15.1:0/552023604 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c38013370 con 0x7f9c4806f810 2024-06-18T02:36:57.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:57.093+0000 7f9c4cecc640 1 -- 172.21.15.1:0/552023604 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c4810fd80 con 0x7f9c4806f810 2024-06-18T02:36:57.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:57.094+0000 7f9c4cecc640 1 -- 172.21.15.1:0/552023604 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c48070f50 con 0x7f9c4806f810 2024-06-18T02:36:57.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:57.094+0000 7f9c3effd640 1 -- 172.21.15.1:0/552023604 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7f9c38012070 con 0x7f9c4806f810 2024-06-18T02:36:57.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:57.095+0000 7f9c3effd640 1 --2- 172.21.15.1:0/552023604 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f9c34065430 0x7f9c340678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:36:57.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:57.095+0000 7f9c3effd640 1 -- 172.21.15.1:0/552023604 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f9c380820c0 con 0x7f9c4806f810 2024-06-18T02:36:57.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:57.095+0000 7f9c3ffff640 1 --2- 172.21.15.1:0/552023604 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f9c34065430 0x7f9c340678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:36:57.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:57.095+0000 7f9c3ffff640 1 --2- 172.21.15.1:0/552023604 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f9c34065430 0x7f9c340678e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f9c30009f80 tx=0x7f9c30009da0 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:36:57.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:57.099+0000 7f9c3effd640 1 -- 172.21.15.1:0/552023604 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f9c38050570 con 0x7f9c4806f810 2024-06-18T02:36:57.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:57.307+0000 7f9c4cecc640 1 -- 172.21.15.1:0/552023604 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f9c48110c80 con 0x7f9c4806f810 2024-06-18T02:36:57.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:57.307+0000 7f9c3effd640 1 -- 172.21.15.1:0/552023604 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f9c38053b90 con 0x7f9c4806f810 2024-06-18T02:36:57.307 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:36:57.307 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-18T02:36:57.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:57.309+0000 7f9c4cecc640 1 -- 172.21.15.1:0/552023604 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f9c34065430 msgr2=0x7f9c340678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:36:57.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:57.309+0000 7f9c4cecc640 1 --2- 172.21.15.1:0/552023604 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f9c34065430 0x7f9c340678e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f9c30009f80 tx=0x7f9c30009da0 comp rx=0 tx=0).stop 2024-06-18T02:36:57.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:57.309+0000 7f9c4cecc640 1 -- 172.21.15.1:0/552023604 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c4806f810 msgr2=0x7f9c481155e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:36:57.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:57.309+0000 7f9c4cecc640 1 --2- 172.21.15.1:0/552023604 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c4806f810 0x7f9c481155e0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f9c3800b790 tx=0x7f9c380091d0 comp rx=0 tx=0).stop 2024-06-18T02:36:57.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:57.310+0000 7f9c4cecc640 1 -- 172.21.15.1:0/552023604 shutdown_connections 2024-06-18T02:36:57.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:57.310+0000 7f9c4cecc640 1 --2- 172.21.15.1:0/552023604 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f9c34065430 0x7f9c340678e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:57.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:57.310+0000 7f9c4cecc640 1 --2- 172.21.15.1:0/552023604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c48070120 0x7f9c48115b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:57.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:57.310+0000 7f9c4cecc640 1 --2- 172.21.15.1:0/552023604 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c4806f810 0x7f9c481155e0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:36:57.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:57.310+0000 7f9c4cecc640 1 -- 172.21.15.1:0/552023604 >> 172.21.15.1:0/552023604 conn(0x7f9c480f7e50 msgr2=0x7f9c480f9380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:36:57.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:57.310+0000 7f9c4cecc640 1 -- 172.21.15.1:0/552023604 shutdown_connections 2024-06-18T02:36:57.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:36:57.310+0000 7f9c4cecc640 1 -- 172.21.15.1:0/552023604 wait complete. 2024-06-18T02:36:57.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:36:57 smithi001 ceph-mon[29099]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:36:57.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:36:57 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/552023604' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:36:57.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:36:57 smithi028 ceph-mon[35823]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:36:57.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:36:57 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/552023604' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:36:59.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:36:59 smithi028 ceph-mon[35823]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:37:00.127 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:36:59 smithi001 ceph-mon[29099]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:37:00.909 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T02:37:01.186 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:37:01.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:37:01 smithi001 ceph-mon[29099]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:37:01.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:37:01 smithi028 ceph-mon[35823]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:37:03.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:02.987+0000 7fd2f75c1640 1 -- 172.21.15.1:0/3132258305 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2f01022c0 msgr2=0x7fd2f0102690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:37:03.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:02.987+0000 7fd2f75c1640 1 --2- 172.21.15.1:0/3132258305 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2f01022c0 0x7fd2f0102690 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec0048d0 tx=0x7fd2ec009da0 comp rx=0 tx=0).stop 2024-06-18T02:37:03.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:02.987+0000 7fd2f75c1640 1 -- 172.21.15.1:0/3132258305 shutdown_connections 2024-06-18T02:37:03.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:02.987+0000 7fd2f75c1640 1 --2- 172.21.15.1:0/3132258305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2f0102c60 0x7fd2f00fc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:37:03.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:02.987+0000 7fd2f75c1640 1 --2- 172.21.15.1:0/3132258305 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2f01022c0 0x7fd2f0102690 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:37:03.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:02.987+0000 7fd2f75c1640 1 -- 172.21.15.1:0/3132258305 >> 172.21.15.1:0/3132258305 conn(0x7fd2f00f7e50 msgr2=0x7fd2f00fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:37:03.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:02.988+0000 7fd2f75c1640 1 -- 172.21.15.1:0/3132258305 shutdown_connections 2024-06-18T02:37:03.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:02.988+0000 7fd2f75c1640 1 -- 172.21.15.1:0/3132258305 wait complete. 2024-06-18T02:37:03.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:02.988+0000 7fd2f75c1640 1 Processor -- start 2024-06-18T02:37:03.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:02.988+0000 7fd2f75c1640 1 -- start start 2024-06-18T02:37:03.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:02.988+0000 7fd2f75c1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2f01022c0 0x7fd2f006f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:37:03.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:02.988+0000 7fd2f75c1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2f0102c60 0x7fd2f006fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:37:03.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:02.988+0000 7fd2f75c1640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f00701f0 con 0x7fd2f01022c0 2024-06-18T02:37:03.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:02.988+0000 7fd2f75c1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f0070360 con 0x7fd2f0102c60 2024-06-18T02:37:03.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:02.988+0000 7fd2f535d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2f01022c0 0x7fd2f006f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:37:03.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:02.988+0000 7fd2f535d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2f01022c0 0x7fd2f006f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53736/0 (socket says 172.21.15.1:53736) 2024-06-18T02:37:03.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:02.988+0000 7fd2f535d640 1 -- 172.21.15.1:0/3035497835 learned_addr learned my addr 172.21.15.1:0/3035497835 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:37:03.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:02.989+0000 7fd2f535d640 1 -- 172.21.15.1:0/3035497835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2f0102c60 msgr2=0x7fd2f006fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:37:03.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:02.989+0000 7fd2f4b5c640 1 --2- 172.21.15.1:0/3035497835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2f0102c60 0x7fd2f006fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:37:03.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:02.989+0000 7fd2f535d640 1 --2- 172.21.15.1:0/3035497835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2f0102c60 0x7fd2f006fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:37:03.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:02.989+0000 7fd2f535d640 1 -- 172.21.15.1:0/3035497835 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2ec005040 con 0x7fd2f01022c0 2024-06-18T02:37:03.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:02.989+0000 7fd2f4b5c640 1 --2- 172.21.15.1:0/3035497835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2f0102c60 0x7fd2f006fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-18T02:37:03.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:02.989+0000 7fd2f535d640 1 --2- 172.21.15.1:0/3035497835 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2f01022c0 0x7fd2f006f770 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec003080 tx=0x7fd2ec009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:37:03.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:02.989+0000 7fd2e5ffb640 1 -- 172.21.15.1:0/3035497835 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd2ec038070 con 0x7fd2f01022c0 2024-06-18T02:37:03.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:02.989+0000 7fd2e5ffb640 1 -- 172.21.15.1:0/3035497835 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd2ec008ee0 con 0x7fd2f01022c0 2024-06-18T02:37:03.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:02.989+0000 7fd2f75c1640 1 -- 172.21.15.1:0/3035497835 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2f0070590 con 0x7fd2f01022c0 2024-06-18T02:37:03.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:02.989+0000 7fd2e5ffb640 1 -- 172.21.15.1:0/3035497835 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd2ec03d870 con 0x7fd2f01022c0 2024-06-18T02:37:03.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:02.989+0000 7fd2f75c1640 1 -- 172.21.15.1:0/3035497835 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2f01199b0 con 0x7fd2f01022c0 2024-06-18T02:37:03.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:02.990+0000 7fd2f75c1640 1 -- 172.21.15.1:0/3035497835 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2f00fda30 con 0x7fd2f01022c0 2024-06-18T02:37:03.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:02.991+0000 7fd2e5ffb640 1 -- 172.21.15.1:0/3035497835 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7fd2ec030020 con 0x7fd2f01022c0 2024-06-18T02:37:03.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:02.991+0000 7fd2e5ffb640 1 --2- 172.21.15.1:0/3035497835 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fd2dc065430 0x7fd2dc0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:37:03.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:02.991+0000 7fd2e5ffb640 1 -- 172.21.15.1:0/3035497835 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fd2ec0a2230 con 0x7fd2f01022c0 2024-06-18T02:37:03.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:02.991+0000 7fd2f4b5c640 1 --2- 172.21.15.1:0/3035497835 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fd2dc065430 0x7fd2dc0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:37:03.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:02.992+0000 7fd2f4b5c640 1 --2- 172.21.15.1:0/3035497835 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fd2dc065430 0x7fd2dc0678e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd2e0007fd0 tx=0x7fd2e0009040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:37:03.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:02.996+0000 7fd2e5ffb640 1 -- 172.21.15.1:0/3035497835 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fd2ec070760 con 0x7fd2f01022c0 2024-06-18T02:37:03.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:03.203+0000 7fd2f75c1640 1 -- 172.21.15.1:0/3035497835 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd2f00fbab0 con 0x7fd2f01022c0 2024-06-18T02:37:03.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:03.203+0000 7fd2e5ffb640 1 -- 172.21.15.1:0/3035497835 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fd2ec073d80 con 0x7fd2f01022c0 2024-06-18T02:37:03.205 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:37:03.205 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-18T02:37:03.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:03.205+0000 7fd2f75c1640 1 -- 172.21.15.1:0/3035497835 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fd2dc065430 msgr2=0x7fd2dc0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:37:03.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:03.205+0000 7fd2f75c1640 1 --2- 172.21.15.1:0/3035497835 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fd2dc065430 0x7fd2dc0678e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd2e0007fd0 tx=0x7fd2e0009040 comp rx=0 tx=0).stop 2024-06-18T02:37:03.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:03.205+0000 7fd2f75c1640 1 -- 172.21.15.1:0/3035497835 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2f01022c0 msgr2=0x7fd2f006f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:37:03.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:03.205+0000 7fd2f75c1640 1 --2- 172.21.15.1:0/3035497835 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2f01022c0 0x7fd2f006f770 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec003080 tx=0x7fd2ec009da0 comp rx=0 tx=0).stop 2024-06-18T02:37:03.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:03.205+0000 7fd2f75c1640 1 -- 172.21.15.1:0/3035497835 shutdown_connections 2024-06-18T02:37:03.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:03.205+0000 7fd2f75c1640 1 --2- 172.21.15.1:0/3035497835 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fd2dc065430 0x7fd2dc0678e0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:37:03.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:03.205+0000 7fd2f75c1640 1 --2- 172.21.15.1:0/3035497835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2f0102c60 0x7fd2f006fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:37:03.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:03.205+0000 7fd2f75c1640 1 --2- 172.21.15.1:0/3035497835 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd2f01022c0 0x7fd2f006f770 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:37:03.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:03.205+0000 7fd2f75c1640 1 -- 172.21.15.1:0/3035497835 >> 172.21.15.1:0/3035497835 conn(0x7fd2f00f7e50 msgr2=0x7fd2f00f8de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:37:03.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:03.205+0000 7fd2f75c1640 1 -- 172.21.15.1:0/3035497835 shutdown_connections 2024-06-18T02:37:03.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:03.206+0000 7fd2f75c1640 1 -- 172.21.15.1:0/3035497835 wait complete. 2024-06-18T02:37:03.849 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:37:03 smithi001 ceph-mon[29099]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:37:03.849 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:37:03 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/3035497835' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:37:03.989 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:37:03 smithi028 ceph-mon[35823]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:37:03.989 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:37:03 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/3035497835' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:37:05.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:37:05 smithi028 ceph-mon[35823]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:37:06.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:37:05 smithi001 ceph-mon[29099]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:37:06.876 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T02:37:06.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:37:06 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:37:07.127 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:37:06 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:37:07.152 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:37:07.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:37:07 smithi001 ceph-mon[29099]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:37:07.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:37:07 smithi028 ceph-mon[35823]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:37:08.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:08.842+0000 7f95b4288640 1 -- 172.21.15.1:0/2431919048 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95ac0fb570 msgr2=0x7f95ac0fb940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:37:08.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:08.842+0000 7f95b4288640 1 --2- 172.21.15.1:0/2431919048 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95ac0fb570 0x7f95ac0fb940 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f95a40048d0 tx=0x7f95a4009da0 comp rx=0 tx=0).stop 2024-06-18T02:37:08.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:08.842+0000 7f95b4288640 1 -- 172.21.15.1:0/2431919048 shutdown_connections 2024-06-18T02:37:08.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:08.842+0000 7f95b4288640 1 --2- 172.21.15.1:0/2431919048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95ac0fbe80 0x7f95ac0fcd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:37:08.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:08.842+0000 7f95b4288640 1 --2- 172.21.15.1:0/2431919048 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95ac0fb570 0x7f95ac0fb940 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:37:08.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:08.842+0000 7f95b4288640 1 -- 172.21.15.1:0/2431919048 >> 172.21.15.1:0/2431919048 conn(0x7f95ac072110 msgr2=0x7f95ac072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:37:08.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:08.842+0000 7f95b4288640 1 -- 172.21.15.1:0/2431919048 shutdown_connections 2024-06-18T02:37:08.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:08.842+0000 7f95b4288640 1 -- 172.21.15.1:0/2431919048 wait complete. 2024-06-18T02:37:08.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:08.843+0000 7f95b4288640 1 Processor -- start 2024-06-18T02:37:08.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:08.843+0000 7f95b4288640 1 -- start start 2024-06-18T02:37:08.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:08.843+0000 7f95b4288640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95ac0fb570 0x7f95ac115230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:37:08.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:08.843+0000 7f95b4288640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95ac0fbe80 0x7f95ac115770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:37:08.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:08.843+0000 7f95b4288640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95ac115d70 con 0x7f95ac0fb570 2024-06-18T02:37:08.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:08.843+0000 7f95b4288640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95ac10f1b0 con 0x7f95ac0fbe80 2024-06-18T02:37:08.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:08.843+0000 7f95b2024640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95ac0fb570 0x7f95ac115230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:37:08.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:08.843+0000 7f95b2024640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95ac0fb570 0x7f95ac115230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39932/0 (socket says 172.21.15.1:39932) 2024-06-18T02:37:08.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:08.843+0000 7f95b2024640 1 -- 172.21.15.1:0/1083663061 learned_addr learned my addr 172.21.15.1:0/1083663061 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:37:08.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:08.843+0000 7f95b1823640 1 --2- 172.21.15.1:0/1083663061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95ac0fbe80 0x7f95ac115770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:37:08.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:08.843+0000 7f95b2024640 1 -- 172.21.15.1:0/1083663061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95ac0fbe80 msgr2=0x7f95ac115770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:37:08.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:08.843+0000 7f95b2024640 1 --2- 172.21.15.1:0/1083663061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95ac0fbe80 0x7f95ac115770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:37:08.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:08.843+0000 7f95b2024640 1 -- 172.21.15.1:0/1083663061 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95a4005040 con 0x7f95ac0fb570 2024-06-18T02:37:08.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:08.843+0000 7f95b2024640 1 --2- 172.21.15.1:0/1083663061 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95ac0fb570 0x7f95ac115230 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f95a40043d0 tx=0x7f95a4009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:37:08.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:08.844+0000 7f95aaffd640 1 -- 172.21.15.1:0/1083663061 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f95a4038070 con 0x7f95ac0fb570 2024-06-18T02:37:08.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:08.844+0000 7f95b4288640 1 -- 172.21.15.1:0/1083663061 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95ac10f430 con 0x7f95ac0fb570 2024-06-18T02:37:08.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:08.844+0000 7f95aaffd640 1 -- 172.21.15.1:0/1083663061 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f95a4032970 con 0x7f95ac0fb570 2024-06-18T02:37:08.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:08.844+0000 7f95aaffd640 1 -- 172.21.15.1:0/1083663061 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f95a403da60 con 0x7f95ac0fb570 2024-06-18T02:37:08.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:08.844+0000 7f95b4288640 1 -- 172.21.15.1:0/1083663061 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95ac10f970 con 0x7f95ac0fb570 2024-06-18T02:37:08.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:08.845+0000 7f95b4288640 1 -- 172.21.15.1:0/1083663061 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95ac0fc920 con 0x7f95ac0fb570 2024-06-18T02:37:08.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:08.845+0000 7f95aaffd640 1 -- 172.21.15.1:0/1083663061 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7f95a4030020 con 0x7f95ac0fb570 2024-06-18T02:37:08.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:08.846+0000 7f95aaffd640 1 --2- 172.21.15.1:0/1083663061 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f9594065430 0x7f95940678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:37:08.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:08.846+0000 7f95aaffd640 1 -- 172.21.15.1:0/1083663061 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f95a40a26c0 con 0x7f95ac0fb570 2024-06-18T02:37:08.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:08.846+0000 7f95b1823640 1 --2- 172.21.15.1:0/1083663061 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f9594065430 0x7f95940678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:37:08.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:08.846+0000 7f95b1823640 1 --2- 172.21.15.1:0/1083663061 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f9594065430 0x7f95940678e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f95ac110af0 tx=0x7f95a000b040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:37:08.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:08.850+0000 7f95aaffd640 1 -- 172.21.15.1:0/1083663061 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f95a4070b70 con 0x7f95ac0fb570 2024-06-18T02:37:09.061 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:37:09.061 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-18T02:37:09.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:09.060+0000 7f95b4288640 1 -- 172.21.15.1:0/1083663061 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f95ac1108a0 con 0x7f95ac0fb570 2024-06-18T02:37:09.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:09.060+0000 7f95aaffd640 1 -- 172.21.15.1:0/1083663061 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f95a4074190 con 0x7f95ac0fb570 2024-06-18T02:37:09.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:09.062+0000 7f95b4288640 1 -- 172.21.15.1:0/1083663061 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f9594065430 msgr2=0x7f95940678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:37:09.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:09.062+0000 7f95b4288640 1 --2- 172.21.15.1:0/1083663061 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f9594065430 0x7f95940678e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f95ac110af0 tx=0x7f95a000b040 comp rx=0 tx=0).stop 2024-06-18T02:37:09.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:09.062+0000 7f95b4288640 1 -- 172.21.15.1:0/1083663061 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95ac0fb570 msgr2=0x7f95ac115230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:37:09.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:09.062+0000 7f95b4288640 1 --2- 172.21.15.1:0/1083663061 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95ac0fb570 0x7f95ac115230 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f95a40043d0 tx=0x7f95a4009da0 comp rx=0 tx=0).stop 2024-06-18T02:37:09.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:09.062+0000 7f95b4288640 1 -- 172.21.15.1:0/1083663061 shutdown_connections 2024-06-18T02:37:09.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:09.062+0000 7f95b4288640 1 --2- 172.21.15.1:0/1083663061 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f9594065430 0x7f95940678e0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:37:09.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:09.062+0000 7f95b4288640 1 --2- 172.21.15.1:0/1083663061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95ac0fbe80 0x7f95ac115770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:37:09.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:09.062+0000 7f95b4288640 1 --2- 172.21.15.1:0/1083663061 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95ac0fb570 0x7f95ac115230 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:37:09.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:09.062+0000 7f95b4288640 1 -- 172.21.15.1:0/1083663061 >> 172.21.15.1:0/1083663061 conn(0x7f95ac072110 msgr2=0x7f95ac0f85d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:37:09.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:09.062+0000 7f95b4288640 1 -- 172.21.15.1:0/1083663061 shutdown_connections 2024-06-18T02:37:09.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:09.062+0000 7f95b4288640 1 -- 172.21.15.1:0/1083663061 wait complete. 2024-06-18T02:37:09.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:37:09 smithi028 ceph-mon[35823]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:37:09.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:37:09 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/1083663061' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:37:10.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:37:09 smithi001 ceph-mon[29099]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:37:10.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:37:09 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/1083663061' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:37:11.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:37:11 smithi028 ceph-mon[35823]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:37:12.127 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:37:11 smithi001 ceph-mon[29099]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:37:12.686 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T02:37:12.968 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:37:13.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:37:13 smithi028 ceph-mon[35823]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:37:13.997 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:37:13 smithi001 ceph-mon[29099]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:37:14.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:14.499+0000 7f4fa8d37640 1 -- 172.21.15.1:0/418858898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fa40650a0 msgr2=0x7f4fa4065470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:37:14.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:14.499+0000 7f4fa8d37640 1 --2- 172.21.15.1:0/418858898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fa40650a0 0x7f4fa4065470 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c0048d0 tx=0x7f4f9c009da0 comp rx=0 tx=0).stop 2024-06-18T02:37:14.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:14.499+0000 7f4fa8d37640 1 -- 172.21.15.1:0/418858898 shutdown_connections 2024-06-18T02:37:14.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:14.499+0000 7f4fa8d37640 1 --2- 172.21.15.1:0/418858898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fa40fdeb0 0x7f4fa40fe2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:37:14.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:14.499+0000 7f4fa8d37640 1 --2- 172.21.15.1:0/418858898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fa40650a0 0x7f4fa4065470 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:37:14.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:14.499+0000 7f4fa8d37640 1 -- 172.21.15.1:0/418858898 >> 172.21.15.1:0/418858898 conn(0x7f4fa4072110 msgr2=0x7f4fa4072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:37:14.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:14.499+0000 7f4fa8d37640 1 -- 172.21.15.1:0/418858898 shutdown_connections 2024-06-18T02:37:14.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:14.499+0000 7f4fa8d37640 1 -- 172.21.15.1:0/418858898 wait complete. 2024-06-18T02:37:14.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:14.499+0000 7f4fa8d37640 1 Processor -- start 2024-06-18T02:37:14.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:14.499+0000 7f4fa8d37640 1 -- start start 2024-06-18T02:37:14.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:14.500+0000 7f4fa8d37640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fa40650a0 0x7f4fa4113080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:37:14.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:14.500+0000 7f4fa8d37640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fa40fdeb0 0x7f4fa41135c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:37:14.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:14.500+0000 7f4fa8d37640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa4113c50 con 0x7f4fa40fdeb0 2024-06-18T02:37:14.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:14.500+0000 7f4fa8d37640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fa410d000 con 0x7f4fa40650a0 2024-06-18T02:37:14.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:14.500+0000 7f4fa1d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fa40fdeb0 0x7f4fa41135c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:37:14.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:14.500+0000 7f4fa1d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fa40fdeb0 0x7f4fa41135c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45288/0 (socket says 172.21.15.1:45288) 2024-06-18T02:37:14.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:14.500+0000 7f4fa1d9b640 1 -- 172.21.15.1:0/1668806676 learned_addr learned my addr 172.21.15.1:0/1668806676 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:37:14.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:14.500+0000 7f4fa259c640 1 --2- 172.21.15.1:0/1668806676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fa40650a0 0x7f4fa4113080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:37:14.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:14.500+0000 7f4fa1d9b640 1 -- 172.21.15.1:0/1668806676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fa40650a0 msgr2=0x7f4fa4113080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:37:14.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:14.500+0000 7f4fa1d9b640 1 --2- 172.21.15.1:0/1668806676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fa40650a0 0x7f4fa4113080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:37:14.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:14.500+0000 7f4fa1d9b640 1 -- 172.21.15.1:0/1668806676 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f9c005040 con 0x7f4fa40fdeb0 2024-06-18T02:37:14.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:14.500+0000 7f4fa1d9b640 1 --2- 172.21.15.1:0/1668806676 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fa40fdeb0 0x7f4fa41135c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f4f8c00b790 tx=0x7f4f8c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:37:14.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:14.500+0000 7f4f9affd640 1 -- 172.21.15.1:0/1668806676 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f8c00c700 con 0x7f4fa40fdeb0 2024-06-18T02:37:14.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:14.500+0000 7f4fa8d37640 1 -- 172.21.15.1:0/1668806676 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fa410d2e0 con 0x7f4fa40fdeb0 2024-06-18T02:37:14.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:14.501+0000 7f4f9affd640 1 -- 172.21.15.1:0/1668806676 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4f8c00cd30 con 0x7f4fa40fdeb0 2024-06-18T02:37:14.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:14.501+0000 7f4f9affd640 1 -- 172.21.15.1:0/1668806676 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f8c013370 con 0x7f4fa40fdeb0 2024-06-18T02:37:14.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:14.501+0000 7f4fa8d37640 1 -- 172.21.15.1:0/1668806676 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fa410d800 con 0x7f4fa40fdeb0 2024-06-18T02:37:14.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:14.502+0000 7f4fa8d37640 1 -- 172.21.15.1:0/1668806676 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fa40fda50 con 0x7f4fa40fdeb0 2024-06-18T02:37:14.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:14.502+0000 7f4f9affd640 1 -- 172.21.15.1:0/1668806676 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7f4f8c012070 con 0x7f4fa40fdeb0 2024-06-18T02:37:14.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:14.502+0000 7f4f9affd640 1 --2- 172.21.15.1:0/1668806676 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f4f90065430 0x7f4f900678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:37:14.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:14.503+0000 7f4f9affd640 1 -- 172.21.15.1:0/1668806676 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f4f8c081e60 con 0x7f4fa40fdeb0 2024-06-18T02:37:14.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:14.503+0000 7f4fa259c640 1 --2- 172.21.15.1:0/1668806676 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f4f90065430 0x7f4f900678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:37:14.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:14.503+0000 7f4fa259c640 1 --2- 172.21.15.1:0/1668806676 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f4f90065430 0x7f4f900678e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c032430 tx=0x7f4f9c035040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:37:14.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:14.507+0000 7f4f9affd640 1 -- 172.21.15.1:0/1668806676 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f4f8c050270 con 0x7f4fa40fdeb0 2024-06-18T02:37:14.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:14.717+0000 7f4fa8d37640 1 -- 172.21.15.1:0/1668806676 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4fa410e720 con 0x7f4fa40fdeb0 2024-06-18T02:37:14.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:14.718+0000 7f4f9affd640 1 -- 172.21.15.1:0/1668806676 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f4f8c053890 con 0x7f4fa40fdeb0 2024-06-18T02:37:14.718 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:37:14.719 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-18T02:37:14.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:14.720+0000 7f4fa8d37640 1 -- 172.21.15.1:0/1668806676 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f4f90065430 msgr2=0x7f4f900678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:37:14.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:14.720+0000 7f4fa8d37640 1 --2- 172.21.15.1:0/1668806676 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f4f90065430 0x7f4f900678e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c032430 tx=0x7f4f9c035040 comp rx=0 tx=0).stop 2024-06-18T02:37:14.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:14.720+0000 7f4fa8d37640 1 -- 172.21.15.1:0/1668806676 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fa40fdeb0 msgr2=0x7f4fa41135c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:37:14.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:14.720+0000 7f4fa8d37640 1 --2- 172.21.15.1:0/1668806676 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fa40fdeb0 0x7f4fa41135c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f4f8c00b790 tx=0x7f4f8c0091d0 comp rx=0 tx=0).stop 2024-06-18T02:37:14.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:14.720+0000 7f4fa8d37640 1 -- 172.21.15.1:0/1668806676 shutdown_connections 2024-06-18T02:37:14.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:14.720+0000 7f4fa8d37640 1 --2- 172.21.15.1:0/1668806676 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f4f90065430 0x7f4f900678e0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:37:14.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:14.720+0000 7f4fa8d37640 1 --2- 172.21.15.1:0/1668806676 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4fa40fdeb0 0x7f4fa41135c0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:37:14.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:14.720+0000 7f4fa8d37640 1 --2- 172.21.15.1:0/1668806676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fa40650a0 0x7f4fa4113080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:37:14.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:14.720+0000 7f4fa8d37640 1 -- 172.21.15.1:0/1668806676 >> 172.21.15.1:0/1668806676 conn(0x7f4fa4072110 msgr2=0x7f4fa4100fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:37:14.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:14.720+0000 7f4fa8d37640 1 -- 172.21.15.1:0/1668806676 shutdown_connections 2024-06-18T02:37:14.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:14.720+0000 7f4fa8d37640 1 -- 172.21.15.1:0/1668806676 wait complete. 2024-06-18T02:37:15.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:37:15 smithi028 ceph-mon[35823]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:37:15.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:37:15 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/1668806676' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:37:16.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:37:15 smithi001 ceph-mon[29099]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:37:16.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:37:15 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/1668806676' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:37:17.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:37:17 smithi028 ceph-mon[35823]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:37:18.127 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:37:17 smithi001 ceph-mon[29099]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:37:18.364 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T02:37:18.642 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:37:19.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:37:19 smithi028 ceph-mon[35823]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:37:20.017 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:37:19 smithi001 ceph-mon[29099]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:37:20.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:20.236+0000 7f3ad17f5640 1 -- 172.21.15.1:0/921891239 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3acc065030 msgr2=0x7f3acc065400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:37:20.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:20.236+0000 7f3ad17f5640 1 --2- 172.21.15.1:0/921891239 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3acc065030 0x7f3acc065400 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f3abc0048d0 tx=0x7f3abc009da0 comp rx=0 tx=0).stop 2024-06-18T02:37:20.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:20.236+0000 7f3ad17f5640 1 -- 172.21.15.1:0/921891239 shutdown_connections 2024-06-18T02:37:20.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:20.236+0000 7f3ad17f5640 1 --2- 172.21.15.1:0/921891239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3acc0fdea0 0x7f3acc0fe310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:37:20.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:20.236+0000 7f3ad17f5640 1 --2- 172.21.15.1:0/921891239 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3acc065030 0x7f3acc065400 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:37:20.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:20.236+0000 7f3ad17f5640 1 -- 172.21.15.1:0/921891239 >> 172.21.15.1:0/921891239 conn(0x7f3acc072110 msgr2=0x7f3acc072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:37:20.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:20.236+0000 7f3ad17f5640 1 -- 172.21.15.1:0/921891239 shutdown_connections 2024-06-18T02:37:20.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:20.236+0000 7f3ad17f5640 1 -- 172.21.15.1:0/921891239 wait complete. 2024-06-18T02:37:20.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:20.237+0000 7f3ad17f5640 1 Processor -- start 2024-06-18T02:37:20.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:20.237+0000 7f3ad17f5640 1 -- start start 2024-06-18T02:37:20.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:20.237+0000 7f3ad17f5640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3acc065030 0x7f3acc113130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:37:20.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:20.237+0000 7f3ad17f5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3acc0fdea0 0x7f3acc113670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:37:20.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:20.237+0000 7f3ad17f5640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3acc113d00 con 0x7f3acc065030 2024-06-18T02:37:20.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:20.237+0000 7f3ad17f5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3acc10d0b0 con 0x7f3acc0fdea0 2024-06-18T02:37:20.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:20.238+0000 7f3acaffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3acc065030 0x7f3acc113130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:37:20.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:20.238+0000 7f3acaffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3acc065030 0x7f3acc113130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45312/0 (socket says 172.21.15.1:45312) 2024-06-18T02:37:20.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:20.238+0000 7f3acaffd640 1 -- 172.21.15.1:0/1922742554 learned_addr learned my addr 172.21.15.1:0/1922742554 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:37:20.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:20.238+0000 7f3aca7fc640 1 --2- 172.21.15.1:0/1922742554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3acc0fdea0 0x7f3acc113670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:37:20.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:20.238+0000 7f3acaffd640 1 -- 172.21.15.1:0/1922742554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3acc0fdea0 msgr2=0x7f3acc113670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:37:20.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:20.238+0000 7f3acaffd640 1 --2- 172.21.15.1:0/1922742554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3acc0fdea0 0x7f3acc113670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:37:20.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:20.238+0000 7f3acaffd640 1 -- 172.21.15.1:0/1922742554 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3abc005040 con 0x7f3acc065030 2024-06-18T02:37:20.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:20.238+0000 7f3acaffd640 1 --2- 172.21.15.1:0/1922742554 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3acc065030 0x7f3acc113130 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f3abc003080 tx=0x7f3abc009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:37:20.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:20.238+0000 7f3ac37fe640 1 -- 172.21.15.1:0/1922742554 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3abc038070 con 0x7f3acc065030 2024-06-18T02:37:20.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:20.238+0000 7f3ac37fe640 1 -- 172.21.15.1:0/1922742554 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3abc008ee0 con 0x7f3acc065030 2024-06-18T02:37:20.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:20.238+0000 7f3ad17f5640 1 -- 172.21.15.1:0/1922742554 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3acc10d330 con 0x7f3acc065030 2024-06-18T02:37:20.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:20.238+0000 7f3ac37fe640 1 -- 172.21.15.1:0/1922742554 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3abc03d870 con 0x7f3acc065030 2024-06-18T02:37:20.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:20.238+0000 7f3ad17f5640 1 -- 172.21.15.1:0/1922742554 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3acc10d7f0 con 0x7f3acc065030 2024-06-18T02:37:20.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:20.239+0000 7f3ad17f5640 1 -- 172.21.15.1:0/1922742554 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3acc0fda40 con 0x7f3acc065030 2024-06-18T02:37:20.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:20.240+0000 7f3ac37fe640 1 -- 172.21.15.1:0/1922742554 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7f3abc030020 con 0x7f3acc065030 2024-06-18T02:37:20.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:20.240+0000 7f3ac37fe640 1 --2- 172.21.15.1:0/1922742554 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f3ab8065430 0x7f3ab80678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:37:20.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:20.240+0000 7f3ac37fe640 1 -- 172.21.15.1:0/1922742554 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f3abc0a21c0 con 0x7f3acc065030 2024-06-18T02:37:20.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:20.240+0000 7f3aca7fc640 1 --2- 172.21.15.1:0/1922742554 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f3ab8065430 0x7f3ab80678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:37:20.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:20.241+0000 7f3aca7fc640 1 --2- 172.21.15.1:0/1922742554 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f3ab8065430 0x7f3ab80678e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3acc10e9f0 tx=0x7f3ab4008040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:37:20.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:20.245+0000 7f3ac37fe640 1 -- 172.21.15.1:0/1922742554 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f3abc0706f0 con 0x7f3acc065030 2024-06-18T02:37:20.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:20.455+0000 7f3ad17f5640 1 -- 172.21.15.1:0/1922742554 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f3acc072c20 con 0x7f3acc065030 2024-06-18T02:37:20.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:20.455+0000 7f3ac37fe640 1 -- 172.21.15.1:0/1922742554 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f3abc073d10 con 0x7f3acc065030 2024-06-18T02:37:20.455 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:37:20.455 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-18T02:37:20.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:20.457+0000 7f3ad17f5640 1 -- 172.21.15.1:0/1922742554 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f3ab8065430 msgr2=0x7f3ab80678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:37:20.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:20.457+0000 7f3ad17f5640 1 --2- 172.21.15.1:0/1922742554 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f3ab8065430 0x7f3ab80678e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3acc10e9f0 tx=0x7f3ab4008040 comp rx=0 tx=0).stop 2024-06-18T02:37:20.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:20.457+0000 7f3ad17f5640 1 -- 172.21.15.1:0/1922742554 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3acc065030 msgr2=0x7f3acc113130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:37:20.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:20.457+0000 7f3ad17f5640 1 --2- 172.21.15.1:0/1922742554 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3acc065030 0x7f3acc113130 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f3abc003080 tx=0x7f3abc009da0 comp rx=0 tx=0).stop 2024-06-18T02:37:20.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:20.457+0000 7f3ad17f5640 1 -- 172.21.15.1:0/1922742554 shutdown_connections 2024-06-18T02:37:20.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:20.457+0000 7f3ad17f5640 1 --2- 172.21.15.1:0/1922742554 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f3ab8065430 0x7f3ab80678e0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:37:20.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:20.457+0000 7f3ad17f5640 1 --2- 172.21.15.1:0/1922742554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3acc0fdea0 0x7f3acc113670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:37:20.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:20.457+0000 7f3ad17f5640 1 --2- 172.21.15.1:0/1922742554 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3acc065030 0x7f3acc113130 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:37:20.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:20.457+0000 7f3ad17f5640 1 -- 172.21.15.1:0/1922742554 >> 172.21.15.1:0/1922742554 conn(0x7f3acc072110 msgr2=0x7f3acc105120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:37:20.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:20.457+0000 7f3ad17f5640 1 -- 172.21.15.1:0/1922742554 shutdown_connections 2024-06-18T02:37:20.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:20.457+0000 7f3ad17f5640 1 -- 172.21.15.1:0/1922742554 wait complete. 2024-06-18T02:37:20.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:37:20 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/1922742554' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:37:21.076 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:37:20 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/1922742554' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:37:21.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:37:21 smithi028 ceph-mon[35823]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:37:21.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:37:21 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:37:22.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:37:21 smithi001 ceph-mon[29099]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:37:22.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:37:21 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:37:23.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:37:23 smithi001 ceph-mon[29099]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:37:23.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:37:23 smithi028 ceph-mon[35823]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:37:24.102 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T02:37:24.382 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:37:24.811 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:37:24 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:37:24.812 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:37:24 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:37:24.812 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:37:24 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T02:37:24.812 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:37:24 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:37:24.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:37:24 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:37:24.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:37:24 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:37:24.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:37:24 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T02:37:24.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:37:24 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:37:25.776 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:37:25 smithi001 ceph-mon[29099]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:37:25.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:25.965+0000 7f3dfd6fb640 1 -- 172.21.15.1:0/1280394648 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3df80fd2a0 msgr2=0x7f3df80fd670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:37:25.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:25.965+0000 7f3dfd6fb640 1 --2- 172.21.15.1:0/1280394648 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3df80fd2a0 0x7f3df80fd670 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f3de0004900 tx=0x7f3de0009da0 comp rx=0 tx=0).stop 2024-06-18T02:37:25.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:25.965+0000 7f3dfd6fb640 1 -- 172.21.15.1:0/1280394648 shutdown_connections 2024-06-18T02:37:25.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:25.965+0000 7f3dfd6fb640 1 --2- 172.21.15.1:0/1280394648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3df80fdc40 0x7f3df8106310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:37:25.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:25.965+0000 7f3dfd6fb640 1 --2- 172.21.15.1:0/1280394648 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3df80fd2a0 0x7f3df80fd670 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:37:25.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:25.965+0000 7f3dfd6fb640 1 -- 172.21.15.1:0/1280394648 >> 172.21.15.1:0/1280394648 conn(0x7f3df80f6d60 msgr2=0x7f3df80f9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:37:25.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:25.965+0000 7f3dfd6fb640 1 -- 172.21.15.1:0/1280394648 shutdown_connections 2024-06-18T02:37:25.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:25.965+0000 7f3dfd6fb640 1 -- 172.21.15.1:0/1280394648 wait complete. 2024-06-18T02:37:25.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:25.965+0000 7f3dfd6fb640 1 Processor -- start 2024-06-18T02:37:25.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:25.966+0000 7f3dfd6fb640 1 -- start start 2024-06-18T02:37:25.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:25.966+0000 7f3dfd6fb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3df80fd2a0 0x7f3df8113100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:37:25.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:25.966+0000 7f3dfd6fb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3df80fdc40 0x7f3df8113640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:37:25.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:25.966+0000 7f3dfd6fb640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3df8113c40 con 0x7f3df80fdc40 2024-06-18T02:37:25.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:25.966+0000 7f3dfd6fb640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3df810d080 con 0x7f3df80fd2a0 2024-06-18T02:37:25.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:25.966+0000 7f3df67fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3df80fdc40 0x7f3df8113640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:37:25.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:25.966+0000 7f3df67fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3df80fdc40 0x7f3df8113640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54740/0 (socket says 172.21.15.1:54740) 2024-06-18T02:37:25.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:25.966+0000 7f3df67fc640 1 -- 172.21.15.1:0/536009567 learned_addr learned my addr 172.21.15.1:0/536009567 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:37:25.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:25.966+0000 7f3df6ffd640 1 --2- 172.21.15.1:0/536009567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3df80fd2a0 0x7f3df8113100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:37:25.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:25.966+0000 7f3df67fc640 1 -- 172.21.15.1:0/536009567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3df80fd2a0 msgr2=0x7f3df8113100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:37:25.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:25.966+0000 7f3df67fc640 1 --2- 172.21.15.1:0/536009567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3df80fd2a0 0x7f3df8113100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:37:25.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:25.966+0000 7f3df67fc640 1 -- 172.21.15.1:0/536009567 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3de0005040 con 0x7f3df80fdc40 2024-06-18T02:37:25.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:25.966+0000 7f3df67fc640 1 --2- 172.21.15.1:0/536009567 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3df80fdc40 0x7f3df8113640 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f3de8002a80 tx=0x7f3de8004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:37:25.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:25.967+0000 7f3def7fe640 1 -- 172.21.15.1:0/536009567 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3de8007c30 con 0x7f3df80fdc40 2024-06-18T02:37:25.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:25.967+0000 7f3dfd6fb640 1 -- 172.21.15.1:0/536009567 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3df810d360 con 0x7f3df80fdc40 2024-06-18T02:37:25.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:25.967+0000 7f3def7fe640 1 -- 172.21.15.1:0/536009567 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3de800f040 con 0x7f3df80fdc40 2024-06-18T02:37:25.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:25.967+0000 7f3def7fe640 1 -- 172.21.15.1:0/536009567 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3de800db70 con 0x7f3df80fdc40 2024-06-18T02:37:25.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:25.967+0000 7f3dfd6fb640 1 -- 172.21.15.1:0/536009567 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3df810d880 con 0x7f3df80fdc40 2024-06-18T02:37:25.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:25.968+0000 7f3dfd6fb640 1 -- 172.21.15.1:0/536009567 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3df8066030 con 0x7f3df80fdc40 2024-06-18T02:37:25.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:25.968+0000 7f3def7fe640 1 -- 172.21.15.1:0/536009567 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7f3de800dcd0 con 0x7f3df80fdc40 2024-06-18T02:37:25.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:25.968+0000 7f3def7fe640 1 --2- 172.21.15.1:0/536009567 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f3de4065430 0x7f3de40678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:37:25.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:25.969+0000 7f3def7fe640 1 -- 172.21.15.1:0/536009567 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f3de8081d60 con 0x7f3df80fdc40 2024-06-18T02:37:25.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:25.969+0000 7f3df6ffd640 1 --2- 172.21.15.1:0/536009567 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f3de4065430 0x7f3de40678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:37:25.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:25.969+0000 7f3df6ffd640 1 --2- 172.21.15.1:0/536009567 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f3de4065430 0x7f3de40678e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3de002af90 tx=0x7f3de0035040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:37:25.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:25.973+0000 7f3def7fe640 1 -- 172.21.15.1:0/536009567 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f3de804cdf0 con 0x7f3df80fdc40 2024-06-18T02:37:25.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:37:25 smithi028 ceph-mon[35823]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:37:26.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:26.181+0000 7f3dfd6fb640 1 -- 172.21.15.1:0/536009567 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f3df80fa9a0 con 0x7f3df80fdc40 2024-06-18T02:37:26.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:26.181+0000 7f3def7fe640 1 -- 172.21.15.1:0/536009567 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f3de8057e60 con 0x7f3df80fdc40 2024-06-18T02:37:26.181 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:37:26.181 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-18T02:37:26.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:26.183+0000 7f3dfd6fb640 1 -- 172.21.15.1:0/536009567 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f3de4065430 msgr2=0x7f3de40678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:37:26.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:26.183+0000 7f3dfd6fb640 1 --2- 172.21.15.1:0/536009567 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f3de4065430 0x7f3de40678e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3de002af90 tx=0x7f3de0035040 comp rx=0 tx=0).stop 2024-06-18T02:37:26.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:26.183+0000 7f3dfd6fb640 1 -- 172.21.15.1:0/536009567 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3df80fdc40 msgr2=0x7f3df8113640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:37:26.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:26.183+0000 7f3dfd6fb640 1 --2- 172.21.15.1:0/536009567 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3df80fdc40 0x7f3df8113640 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f3de8002a80 tx=0x7f3de8004600 comp rx=0 tx=0).stop 2024-06-18T02:37:26.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:26.183+0000 7f3dfd6fb640 1 -- 172.21.15.1:0/536009567 shutdown_connections 2024-06-18T02:37:26.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:26.183+0000 7f3dfd6fb640 1 --2- 172.21.15.1:0/536009567 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f3de4065430 0x7f3de40678e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:37:26.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:26.183+0000 7f3dfd6fb640 1 --2- 172.21.15.1:0/536009567 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3df80fdc40 0x7f3df8113640 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:37:26.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:26.184+0000 7f3dfd6fb640 1 --2- 172.21.15.1:0/536009567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3df80fd2a0 0x7f3df8113100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:37:26.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:26.184+0000 7f3dfd6fb640 1 -- 172.21.15.1:0/536009567 >> 172.21.15.1:0/536009567 conn(0x7f3df80f6d60 msgr2=0x7f3df80f8260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:37:26.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:26.184+0000 7f3dfd6fb640 1 -- 172.21.15.1:0/536009567 shutdown_connections 2024-06-18T02:37:26.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:26.184+0000 7f3dfd6fb640 1 -- 172.21.15.1:0/536009567 wait complete. 2024-06-18T02:37:26.797 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:37:26 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/536009567' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:37:26.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:37:26 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/536009567' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:37:27.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:37:27 smithi028 ceph-mon[35823]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:37:28.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:37:27 smithi001 ceph-mon[29099]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:37:29.826 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T02:37:29.990 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:37:29 smithi028 ceph-mon[35823]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:37:30.104 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:37:30.127 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:37:29 smithi001 ceph-mon[29099]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:37:31.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:31.678+0000 7fe4efebc640 1 -- 172.21.15.1:0/3666363175 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe4e8101310 msgr2=0x7fe4e81016e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:37:31.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:31.678+0000 7fe4efebc640 1 --2- 172.21.15.1:0/3666363175 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe4e8101310 0x7fe4e81016e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fe4e40048d0 tx=0x7fe4e4009da0 comp rx=0 tx=0).stop 2024-06-18T02:37:31.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:31.678+0000 7fe4efebc640 1 -- 172.21.15.1:0/3666363175 shutdown_connections 2024-06-18T02:37:31.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:31.678+0000 7fe4efebc640 1 --2- 172.21.15.1:0/3666363175 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4e8101cb0 0x7fe4e80fb230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:37:31.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:31.678+0000 7fe4efebc640 1 --2- 172.21.15.1:0/3666363175 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe4e8101310 0x7fe4e81016e0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:37:31.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:31.678+0000 7fe4efebc640 1 -- 172.21.15.1:0/3666363175 >> 172.21.15.1:0/3666363175 conn(0x7fe4e80f6d90 msgr2=0x7fe4e80f91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:37:31.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:31.678+0000 7fe4efebc640 1 -- 172.21.15.1:0/3666363175 shutdown_connections 2024-06-18T02:37:31.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:31.678+0000 7fe4efebc640 1 -- 172.21.15.1:0/3666363175 wait complete. 2024-06-18T02:37:31.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:31.678+0000 7fe4efebc640 1 Processor -- start 2024-06-18T02:37:31.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:31.678+0000 7fe4efebc640 1 -- start start 2024-06-18T02:37:31.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:31.679+0000 7fe4efebc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe4e8101310 0x7fe4e81130c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:37:31.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:31.679+0000 7fe4efebc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4e8101cb0 0x7fe4e8113600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:37:31.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:31.679+0000 7fe4efebc640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e8113c00 con 0x7fe4e8101310 2024-06-18T02:37:31.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:31.679+0000 7fe4efebc640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e810d040 con 0x7fe4e8101cb0 2024-06-18T02:37:31.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:31.679+0000 7fe4edc58640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe4e8101310 0x7fe4e81130c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:37:31.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:31.679+0000 7fe4edc58640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe4e8101310 0x7fe4e81130c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54768/0 (socket says 172.21.15.1:54768) 2024-06-18T02:37:31.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:31.679+0000 7fe4edc58640 1 -- 172.21.15.1:0/3379767260 learned_addr learned my addr 172.21.15.1:0/3379767260 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:37:31.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:31.679+0000 7fe4ed457640 1 --2- 172.21.15.1:0/3379767260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4e8101cb0 0x7fe4e8113600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:37:31.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:31.679+0000 7fe4edc58640 1 -- 172.21.15.1:0/3379767260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4e8101cb0 msgr2=0x7fe4e8113600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:37:31.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:31.679+0000 7fe4edc58640 1 --2- 172.21.15.1:0/3379767260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4e8101cb0 0x7fe4e8113600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:37:31.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:31.679+0000 7fe4edc58640 1 -- 172.21.15.1:0/3379767260 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4e4005040 con 0x7fe4e8101310 2024-06-18T02:37:31.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:31.679+0000 7fe4edc58640 1 --2- 172.21.15.1:0/3379767260 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe4e8101310 0x7fe4e81130c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fe4e40043d0 tx=0x7fe4e4009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:37:31.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:31.679+0000 7fe4de7fc640 1 -- 172.21.15.1:0/3379767260 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe4e4038070 con 0x7fe4e8101310 2024-06-18T02:37:31.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:31.679+0000 7fe4de7fc640 1 -- 172.21.15.1:0/3379767260 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe4e4032970 con 0x7fe4e8101310 2024-06-18T02:37:31.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:31.679+0000 7fe4efebc640 1 -- 172.21.15.1:0/3379767260 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4e810d2c0 con 0x7fe4e8101310 2024-06-18T02:37:31.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:31.680+0000 7fe4de7fc640 1 -- 172.21.15.1:0/3379767260 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe4e403da60 con 0x7fe4e8101310 2024-06-18T02:37:31.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:31.680+0000 7fe4efebc640 1 -- 172.21.15.1:0/3379767260 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4e810d800 con 0x7fe4e8101310 2024-06-18T02:37:31.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:31.681+0000 7fe4efebc640 1 -- 172.21.15.1:0/3379767260 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4e80fa9f0 con 0x7fe4e8101310 2024-06-18T02:37:31.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:31.681+0000 7fe4de7fc640 1 -- 172.21.15.1:0/3379767260 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7fe4e4030020 con 0x7fe4e8101310 2024-06-18T02:37:31.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:31.681+0000 7fe4de7fc640 1 --2- 172.21.15.1:0/3379767260 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fe4d4065430 0x7fe4d40678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:37:31.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:31.682+0000 7fe4ed457640 1 --2- 172.21.15.1:0/3379767260 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fe4d4065430 0x7fe4d40678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:37:31.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:31.682+0000 7fe4de7fc640 1 -- 172.21.15.1:0/3379767260 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fe4e40a2520 con 0x7fe4e8101310 2024-06-18T02:37:31.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:31.682+0000 7fe4ed457640 1 --2- 172.21.15.1:0/3379767260 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fe4d4065430 0x7fe4d40678e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe4e810e980 tx=0x7fe4d800b040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:37:31.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:31.686+0000 7fe4de7fc640 1 -- 172.21.15.1:0/3379767260 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fe4e40709d0 con 0x7fe4e8101310 2024-06-18T02:37:31.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:31.897+0000 7fe4efebc640 1 -- 172.21.15.1:0/3379767260 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe4e8108470 con 0x7fe4e8101310 2024-06-18T02:37:31.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:31.898+0000 7fe4de7fc640 1 -- 172.21.15.1:0/3379767260 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fe4e4073ff0 con 0x7fe4e8101310 2024-06-18T02:37:31.898 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:37:31.898 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-18T02:37:31.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:31.900+0000 7fe4efebc640 1 -- 172.21.15.1:0/3379767260 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fe4d4065430 msgr2=0x7fe4d40678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:37:31.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:31.900+0000 7fe4efebc640 1 --2- 172.21.15.1:0/3379767260 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fe4d4065430 0x7fe4d40678e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe4e810e980 tx=0x7fe4d800b040 comp rx=0 tx=0).stop 2024-06-18T02:37:31.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:31.900+0000 7fe4efebc640 1 -- 172.21.15.1:0/3379767260 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe4e8101310 msgr2=0x7fe4e81130c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:37:31.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:31.900+0000 7fe4efebc640 1 --2- 172.21.15.1:0/3379767260 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe4e8101310 0x7fe4e81130c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fe4e40043d0 tx=0x7fe4e4009da0 comp rx=0 tx=0).stop 2024-06-18T02:37:31.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:31.900+0000 7fe4efebc640 1 -- 172.21.15.1:0/3379767260 shutdown_connections 2024-06-18T02:37:31.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:31.900+0000 7fe4efebc640 1 --2- 172.21.15.1:0/3379767260 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fe4d4065430 0x7fe4d40678e0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:37:31.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:31.900+0000 7fe4efebc640 1 --2- 172.21.15.1:0/3379767260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4e8101cb0 0x7fe4e8113600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:37:31.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:31.900+0000 7fe4efebc640 1 --2- 172.21.15.1:0/3379767260 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe4e8101310 0x7fe4e81130c0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:37:31.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:31.900+0000 7fe4efebc640 1 -- 172.21.15.1:0/3379767260 >> 172.21.15.1:0/3379767260 conn(0x7fe4e80f6d90 msgr2=0x7fe4e80f84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:37:31.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:31.900+0000 7fe4efebc640 1 -- 172.21.15.1:0/3379767260 shutdown_connections 2024-06-18T02:37:31.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:31.900+0000 7fe4efebc640 1 -- 172.21.15.1:0/3379767260 wait complete. 2024-06-18T02:37:31.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:37:31 smithi028 ceph-mon[35823]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:37:32.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:37:31 smithi001 ceph-mon[29099]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:37:32.768 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:37:32 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/3379767260' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:37:32.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:37:32 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/3379767260' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:37:33.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:37:33 smithi001 ceph-mon[29099]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:37:33.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:37:33 smithi028 ceph-mon[35823]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:37:35.503 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T02:37:35.781 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:37:35.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:37:35 smithi001 ceph-mon[29099]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:37:35.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:37:35 smithi028 ceph-mon[35823]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:37:36.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:37:36 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:37:37.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:37:36 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:37:37.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:37.468+0000 7f82f0a61640 1 -- 172.21.15.1:0/4131122796 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82ec101200 msgr2=0x7f82ec1015d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:37:37.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:37.468+0000 7f82f0a61640 1 --2- 172.21.15.1:0/4131122796 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82ec101200 0x7f82ec1015d0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f82dc0048d0 tx=0x7f82dc009da0 comp rx=0 tx=0).stop 2024-06-18T02:37:37.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:37.468+0000 7f82f0a61640 1 -- 172.21.15.1:0/4131122796 shutdown_connections 2024-06-18T02:37:37.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:37.468+0000 7f82f0a61640 1 --2- 172.21.15.1:0/4131122796 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82ec101ba0 0x7f82ec0fb230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:37:37.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:37.468+0000 7f82f0a61640 1 --2- 172.21.15.1:0/4131122796 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82ec101200 0x7f82ec1015d0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:37:37.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:37.468+0000 7f82f0a61640 1 -- 172.21.15.1:0/4131122796 >> 172.21.15.1:0/4131122796 conn(0x7f82ec0f6df0 msgr2=0x7f82ec0f9200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:37:37.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:37.468+0000 7f82f0a61640 1 -- 172.21.15.1:0/4131122796 shutdown_connections 2024-06-18T02:37:37.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:37.468+0000 7f82f0a61640 1 -- 172.21.15.1:0/4131122796 wait complete. 2024-06-18T02:37:37.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:37.469+0000 7f82f0a61640 1 Processor -- start 2024-06-18T02:37:37.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:37.469+0000 7f82f0a61640 1 -- start start 2024-06-18T02:37:37.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:37.469+0000 7f82f0a61640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82ec101200 0x7f82ec10dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:37:37.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:37.469+0000 7f82f0a61640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82ec101ba0 0x7f82ec10e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:37:37.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:37.469+0000 7f82f0a61640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82ec10ec10 con 0x7f82ec101ba0 2024-06-18T02:37:37.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:37.469+0000 7f82f0a61640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82ec10ed80 con 0x7f82ec101200 2024-06-18T02:37:37.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:37.469+0000 7f82e9d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82ec101ba0 0x7f82ec10e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:37:37.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:37.469+0000 7f82e9d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82ec101ba0 0x7f82ec10e530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55622/0 (socket says 172.21.15.1:55622) 2024-06-18T02:37:37.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:37.469+0000 7f82e9d9b640 1 -- 172.21.15.1:0/2741059378 learned_addr learned my addr 172.21.15.1:0/2741059378 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:37:37.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:37.469+0000 7f82ea59c640 1 --2- 172.21.15.1:0/2741059378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82ec101200 0x7f82ec10dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:37:37.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:37.470+0000 7f82e9d9b640 1 -- 172.21.15.1:0/2741059378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82ec101200 msgr2=0x7f82ec10dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:37:37.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:37.470+0000 7f82e9d9b640 1 --2- 172.21.15.1:0/2741059378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82ec101200 0x7f82ec10dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:37:37.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:37.470+0000 7f82e9d9b640 1 -- 172.21.15.1:0/2741059378 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82dc005040 con 0x7f82ec101ba0 2024-06-18T02:37:37.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:37.470+0000 7f82e9d9b640 1 --2- 172.21.15.1:0/2741059378 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82ec101ba0 0x7f82ec10e530 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f82e000b790 tx=0x7f82e00091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:37:37.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:37.470+0000 7f82daffd640 1 -- 172.21.15.1:0/2741059378 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82e000c700 con 0x7f82ec101ba0 2024-06-18T02:37:37.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:37.470+0000 7f82daffd640 1 -- 172.21.15.1:0/2741059378 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f82e000cd30 con 0x7f82ec101ba0 2024-06-18T02:37:37.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:37.470+0000 7f82f0a61640 1 -- 172.21.15.1:0/2741059378 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82ec117060 con 0x7f82ec101ba0 2024-06-18T02:37:37.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:37.470+0000 7f82daffd640 1 -- 172.21.15.1:0/2741059378 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82e0013370 con 0x7f82ec101ba0 2024-06-18T02:37:37.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:37.470+0000 7f82f0a61640 1 -- 172.21.15.1:0/2741059378 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82ec117580 con 0x7f82ec101ba0 2024-06-18T02:37:37.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:37.471+0000 7f82f0a61640 1 -- 172.21.15.1:0/2741059378 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82ec0fc970 con 0x7f82ec101ba0 2024-06-18T02:37:37.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:37.471+0000 7f82daffd640 1 -- 172.21.15.1:0/2741059378 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7f82e0012070 con 0x7f82ec101ba0 2024-06-18T02:37:37.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:37.472+0000 7f82daffd640 1 --2- 172.21.15.1:0/2741059378 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f82d4065430 0x7f82d40678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:37:37.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:37.472+0000 7f82daffd640 1 -- 172.21.15.1:0/2741059378 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f82e00820c0 con 0x7f82ec101ba0 2024-06-18T02:37:37.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:37.472+0000 7f82ea59c640 1 --2- 172.21.15.1:0/2741059378 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f82d4065430 0x7f82d40678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:37:37.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:37.472+0000 7f82ea59c640 1 --2- 172.21.15.1:0/2741059378 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f82d4065430 0x7f82d40678e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f82dc000c00 tx=0x7f82dc035040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:37:37.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:37.476+0000 7f82daffd640 1 -- 172.21.15.1:0/2741059378 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f82e0050570 con 0x7f82ec101ba0 2024-06-18T02:37:37.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:37.692+0000 7f82f0a61640 1 -- 172.21.15.1:0/2741059378 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f82ec04b480 con 0x7f82ec101ba0 2024-06-18T02:37:37.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:37.692+0000 7f82daffd640 1 -- 172.21.15.1:0/2741059378 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f82e0053b90 con 0x7f82ec101ba0 2024-06-18T02:37:37.692 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:37:37.692 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-18T02:37:37.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:37.694+0000 7f82f0a61640 1 -- 172.21.15.1:0/2741059378 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f82d4065430 msgr2=0x7f82d40678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:37:37.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:37.694+0000 7f82f0a61640 1 --2- 172.21.15.1:0/2741059378 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f82d4065430 0x7f82d40678e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f82dc000c00 tx=0x7f82dc035040 comp rx=0 tx=0).stop 2024-06-18T02:37:37.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:37.694+0000 7f82f0a61640 1 -- 172.21.15.1:0/2741059378 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82ec101ba0 msgr2=0x7f82ec10e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:37:37.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:37.694+0000 7f82f0a61640 1 --2- 172.21.15.1:0/2741059378 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82ec101ba0 0x7f82ec10e530 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f82e000b790 tx=0x7f82e00091d0 comp rx=0 tx=0).stop 2024-06-18T02:37:37.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:37.695+0000 7f82f0a61640 1 -- 172.21.15.1:0/2741059378 shutdown_connections 2024-06-18T02:37:37.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:37.695+0000 7f82f0a61640 1 --2- 172.21.15.1:0/2741059378 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f82d4065430 0x7f82d40678e0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:37:37.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:37.695+0000 7f82f0a61640 1 --2- 172.21.15.1:0/2741059378 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82ec101ba0 0x7f82ec10e530 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:37:37.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:37.695+0000 7f82f0a61640 1 --2- 172.21.15.1:0/2741059378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82ec101200 0x7f82ec10dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:37:37.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:37.695+0000 7f82f0a61640 1 -- 172.21.15.1:0/2741059378 >> 172.21.15.1:0/2741059378 conn(0x7f82ec0f6df0 msgr2=0x7f82ec0f7db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:37:37.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:37.695+0000 7f82f0a61640 1 -- 172.21.15.1:0/2741059378 shutdown_connections 2024-06-18T02:37:37.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:37.695+0000 7f82f0a61640 1 -- 172.21.15.1:0/2741059378 wait complete. 2024-06-18T02:37:38.013 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:37:37 smithi001 ceph-mon[29099]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:37:38.013 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:37:37 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/2741059378' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:37:38.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:37:37 smithi028 ceph-mon[35823]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:37:38.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:37:37 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/2741059378' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:37:40.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:37:39 smithi001 ceph-mon[29099]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:37:40.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:37:39 smithi028 ceph-mon[35823]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:37:41.425 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T02:37:41.703 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:37:42.064 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:37:41 smithi001 ceph-mon[29099]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:37:42.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:37:41 smithi028 ceph-mon[35823]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:37:43.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:43.279+0000 7fbafb0f9640 1 -- 172.21.15.1:0/2454939975 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbaf41022c0 msgr2=0x7fbaf4102690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:37:43.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:43.279+0000 7fbafb0f9640 1 --2- 172.21.15.1:0/2454939975 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbaf41022c0 0x7fbaf4102690 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fbae80048d0 tx=0x7fbae8009da0 comp rx=0 tx=0).stop 2024-06-18T02:37:43.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:43.279+0000 7fbafb0f9640 1 -- 172.21.15.1:0/2454939975 shutdown_connections 2024-06-18T02:37:43.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:43.279+0000 7fbafb0f9640 1 --2- 172.21.15.1:0/2454939975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbaf4102c60 0x7fbaf40fc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:37:43.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:43.279+0000 7fbafb0f9640 1 --2- 172.21.15.1:0/2454939975 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbaf41022c0 0x7fbaf4102690 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:37:43.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:43.279+0000 7fbafb0f9640 1 -- 172.21.15.1:0/2454939975 >> 172.21.15.1:0/2454939975 conn(0x7fbaf40f7e50 msgr2=0x7fbaf40fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:37:43.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:43.279+0000 7fbafb0f9640 1 -- 172.21.15.1:0/2454939975 shutdown_connections 2024-06-18T02:37:43.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:43.279+0000 7fbafb0f9640 1 -- 172.21.15.1:0/2454939975 wait complete. 2024-06-18T02:37:43.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:43.279+0000 7fbafb0f9640 1 Processor -- start 2024-06-18T02:37:43.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:43.279+0000 7fbafb0f9640 1 -- start start 2024-06-18T02:37:43.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:43.280+0000 7fbafb0f9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbaf41022c0 0x7fbaf41132f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:37:43.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:43.280+0000 7fbafb0f9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbaf4102c60 0x7fbaf4113830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:37:43.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:43.280+0000 7fbafb0f9640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf4113e30 con 0x7fbaf41022c0 2024-06-18T02:37:43.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:43.280+0000 7fbafb0f9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf410d270 con 0x7fbaf4102c60 2024-06-18T02:37:43.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:43.280+0000 7fbaf8e95640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbaf41022c0 0x7fbaf41132f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:37:43.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:43.280+0000 7fbaf8e95640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbaf41022c0 0x7fbaf41132f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55646/0 (socket says 172.21.15.1:55646) 2024-06-18T02:37:43.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:43.280+0000 7fbaf8e95640 1 -- 172.21.15.1:0/1625474465 learned_addr learned my addr 172.21.15.1:0/1625474465 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:37:43.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:43.280+0000 7fbaf3fff640 1 --2- 172.21.15.1:0/1625474465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbaf4102c60 0x7fbaf4113830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:37:43.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:43.280+0000 7fbaf8e95640 1 -- 172.21.15.1:0/1625474465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbaf4102c60 msgr2=0x7fbaf4113830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:37:43.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:43.280+0000 7fbaf8e95640 1 --2- 172.21.15.1:0/1625474465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbaf4102c60 0x7fbaf4113830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:37:43.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:43.280+0000 7fbaf8e95640 1 -- 172.21.15.1:0/1625474465 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbae8005040 con 0x7fbaf41022c0 2024-06-18T02:37:43.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:43.280+0000 7fbaf3fff640 1 --2- 172.21.15.1:0/1625474465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbaf4102c60 0x7fbaf4113830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-18T02:37:43.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:43.280+0000 7fbaf8e95640 1 --2- 172.21.15.1:0/1625474465 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbaf41022c0 0x7fbaf41132f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fbae8003080 tx=0x7fbae8009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:37:43.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:43.280+0000 7fbaf17fa640 1 -- 172.21.15.1:0/1625474465 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbae8038070 con 0x7fbaf41022c0 2024-06-18T02:37:43.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:43.280+0000 7fbaf17fa640 1 -- 172.21.15.1:0/1625474465 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbae8008ee0 con 0x7fbaf41022c0 2024-06-18T02:37:43.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:43.280+0000 7fbafb0f9640 1 -- 172.21.15.1:0/1625474465 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaf410d4f0 con 0x7fbaf41022c0 2024-06-18T02:37:43.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:43.280+0000 7fbaf17fa640 1 -- 172.21.15.1:0/1625474465 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbae803d870 con 0x7fbaf41022c0 2024-06-18T02:37:43.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:43.280+0000 7fbafb0f9640 1 -- 172.21.15.1:0/1625474465 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaf410da30 con 0x7fbaf41022c0 2024-06-18T02:37:43.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:43.281+0000 7fbafb0f9640 1 -- 172.21.15.1:0/1625474465 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbaf40fda30 con 0x7fbaf41022c0 2024-06-18T02:37:43.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:43.282+0000 7fbaf17fa640 1 -- 172.21.15.1:0/1625474465 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7fbae8030020 con 0x7fbaf41022c0 2024-06-18T02:37:43.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:43.282+0000 7fbaf17fa640 1 --2- 172.21.15.1:0/1625474465 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fbae4065430 0x7fbae40678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:37:43.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:43.282+0000 7fbaf17fa640 1 -- 172.21.15.1:0/1625474465 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fbae80a2140 con 0x7fbaf41022c0 2024-06-18T02:37:43.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:43.282+0000 7fbaf3fff640 1 --2- 172.21.15.1:0/1625474465 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fbae4065430 0x7fbae40678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:37:43.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:43.283+0000 7fbaf3fff640 1 --2- 172.21.15.1:0/1625474465 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fbae4065430 0x7fbae40678e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fbae0007fd0 tx=0x7fbae0009040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:37:43.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:43.286+0000 7fbaf17fa640 1 -- 172.21.15.1:0/1625474465 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fbae8070710 con 0x7fbaf41022c0 2024-06-18T02:37:43.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:43.497+0000 7fbafb0f9640 1 -- 172.21.15.1:0/1625474465 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fbaf4109400 con 0x7fbaf41022c0 2024-06-18T02:37:43.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:43.497+0000 7fbaf17fa640 1 -- 172.21.15.1:0/1625474465 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fbae8073d30 con 0x7fbaf41022c0 2024-06-18T02:37:43.497 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:37:43.498 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-18T02:37:43.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:43.499+0000 7fbafb0f9640 1 -- 172.21.15.1:0/1625474465 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fbae4065430 msgr2=0x7fbae40678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:37:43.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:43.499+0000 7fbafb0f9640 1 --2- 172.21.15.1:0/1625474465 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fbae4065430 0x7fbae40678e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fbae0007fd0 tx=0x7fbae0009040 comp rx=0 tx=0).stop 2024-06-18T02:37:43.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:43.499+0000 7fbafb0f9640 1 -- 172.21.15.1:0/1625474465 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbaf41022c0 msgr2=0x7fbaf41132f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:37:43.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:43.499+0000 7fbafb0f9640 1 --2- 172.21.15.1:0/1625474465 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbaf41022c0 0x7fbaf41132f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fbae8003080 tx=0x7fbae8009da0 comp rx=0 tx=0).stop 2024-06-18T02:37:43.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:43.499+0000 7fbafb0f9640 1 -- 172.21.15.1:0/1625474465 shutdown_connections 2024-06-18T02:37:43.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:43.499+0000 7fbafb0f9640 1 --2- 172.21.15.1:0/1625474465 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fbae4065430 0x7fbae40678e0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:37:43.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:43.499+0000 7fbafb0f9640 1 --2- 172.21.15.1:0/1625474465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbaf4102c60 0x7fbaf4113830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:37:43.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:43.499+0000 7fbafb0f9640 1 --2- 172.21.15.1:0/1625474465 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbaf41022c0 0x7fbaf41132f0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:37:43.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:43.499+0000 7fbafb0f9640 1 -- 172.21.15.1:0/1625474465 >> 172.21.15.1:0/1625474465 conn(0x7fbaf40f7e50 msgr2=0x7fbaf40f8e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:37:43.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:43.499+0000 7fbafb0f9640 1 -- 172.21.15.1:0/1625474465 shutdown_connections 2024-06-18T02:37:43.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:43.499+0000 7fbafb0f9640 1 -- 172.21.15.1:0/1625474465 wait complete. 2024-06-18T02:37:43.801 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:37:43 smithi001 ceph-mon[29099]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:37:43.801 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:37:43 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/1625474465' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:37:44.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:37:43 smithi028 ceph-mon[35823]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:37:44.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:37:43 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/1625474465' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:37:46.127 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:37:45 smithi001 ceph-mon[29099]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:37:46.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:37:45 smithi028 ceph-mon[35823]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:37:47.280 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T02:37:47.556 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:37:47.978 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:37:47 smithi001 ceph-mon[29099]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:37:48.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:37:47 smithi028 ceph-mon[35823]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:37:49.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:49.164+0000 7fb03e844640 1 -- 172.21.15.1:0/4159175987 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0380faa50 msgr2=0x7fb0380faec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:37:49.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:49.164+0000 7fb03e844640 1 --2- 172.21.15.1:0/4159175987 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0380faa50 0x7fb0380faec0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fb02c0048d0 tx=0x7fb02c009da0 comp rx=0 tx=0).stop 2024-06-18T02:37:49.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:49.164+0000 7fb03e844640 1 -- 172.21.15.1:0/4159175987 shutdown_connections 2024-06-18T02:37:49.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:49.164+0000 7fb03e844640 1 --2- 172.21.15.1:0/4159175987 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0380faa50 0x7fb0380faec0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:37:49.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:49.164+0000 7fb03e844640 1 --2- 172.21.15.1:0/4159175987 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0381023e0 0x7fb0381027b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:37:49.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:49.164+0000 7fb03e844640 1 -- 172.21.15.1:0/4159175987 >> 172.21.15.1:0/4159175987 conn(0x7fb038070ab0 msgr2=0x7fb038070eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:37:49.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:49.164+0000 7fb03e844640 1 -- 172.21.15.1:0/4159175987 shutdown_connections 2024-06-18T02:37:49.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:49.164+0000 7fb03e844640 1 -- 172.21.15.1:0/4159175987 wait complete. 2024-06-18T02:37:49.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:49.165+0000 7fb03e844640 1 Processor -- start 2024-06-18T02:37:49.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:49.165+0000 7fb03e844640 1 -- start start 2024-06-18T02:37:49.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:49.165+0000 7fb03e844640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0380faa50 0x7fb038113180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:37:49.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:49.165+0000 7fb03e844640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0381023e0 0x7fb0381136c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:37:49.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:49.165+0000 7fb03e844640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb038113d50 con 0x7fb0380faa50 2024-06-18T02:37:49.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:49.165+0000 7fb03e844640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb03810d100 con 0x7fb0381023e0 2024-06-18T02:37:49.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:49.165+0000 7fb037fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0380faa50 0x7fb038113180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:37:49.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:49.165+0000 7fb037fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0380faa50 0x7fb038113180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60036/0 (socket says 172.21.15.1:60036) 2024-06-18T02:37:49.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:49.165+0000 7fb037fff640 1 -- 172.21.15.1:0/931456159 learned_addr learned my addr 172.21.15.1:0/931456159 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:37:49.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:49.165+0000 7fb037fff640 1 -- 172.21.15.1:0/931456159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0381023e0 msgr2=0x7fb0381136c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:37:49.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:49.165+0000 7fb0377fe640 1 --2- 172.21.15.1:0/931456159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0381023e0 0x7fb0381136c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:37:49.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:49.165+0000 7fb037fff640 1 --2- 172.21.15.1:0/931456159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0381023e0 0x7fb0381136c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:37:49.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:49.165+0000 7fb037fff640 1 -- 172.21.15.1:0/931456159 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb02c005040 con 0x7fb0380faa50 2024-06-18T02:37:49.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:49.166+0000 7fb037fff640 1 --2- 172.21.15.1:0/931456159 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0380faa50 0x7fb038113180 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fb02800b790 tx=0x7fb0280091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:37:49.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:49.166+0000 7fb034ff9640 1 -- 172.21.15.1:0/931456159 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb02800c700 con 0x7fb0380faa50 2024-06-18T02:37:49.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:49.166+0000 7fb03e844640 1 -- 172.21.15.1:0/931456159 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb03810d3e0 con 0x7fb0380faa50 2024-06-18T02:37:49.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:49.166+0000 7fb034ff9640 1 -- 172.21.15.1:0/931456159 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb02800cd30 con 0x7fb0380faa50 2024-06-18T02:37:49.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:49.166+0000 7fb034ff9640 1 -- 172.21.15.1:0/931456159 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb028013370 con 0x7fb0380faa50 2024-06-18T02:37:49.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:49.166+0000 7fb03e844640 1 -- 172.21.15.1:0/931456159 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb03810d900 con 0x7fb0380faa50 2024-06-18T02:37:49.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:49.167+0000 7fb03e844640 1 -- 172.21.15.1:0/931456159 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb038065550 con 0x7fb0380faa50 2024-06-18T02:37:49.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:49.167+0000 7fb034ff9640 1 -- 172.21.15.1:0/931456159 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7fb028012070 con 0x7fb0380faa50 2024-06-18T02:37:49.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:49.168+0000 7fb034ff9640 1 --2- 172.21.15.1:0/931456159 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fb020065430 0x7fb0200678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:37:49.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:49.168+0000 7fb0377fe640 1 --2- 172.21.15.1:0/931456159 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fb020065430 0x7fb0200678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:37:49.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:49.168+0000 7fb034ff9640 1 -- 172.21.15.1:0/931456159 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fb028082160 con 0x7fb0380faa50 2024-06-18T02:37:49.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:49.169+0000 7fb0377fe640 1 --2- 172.21.15.1:0/931456159 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fb020065430 0x7fb0200678e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb02c000c00 tx=0x7fb02c009da0 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:37:49.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:49.172+0000 7fb034ff9640 1 -- 172.21.15.1:0/931456159 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fb028050570 con 0x7fb0380faa50 2024-06-18T02:37:49.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:49.385+0000 7fb03e844640 1 -- 172.21.15.1:0/931456159 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fb0381095b0 con 0x7fb0380faa50 2024-06-18T02:37:49.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:49.385+0000 7fb034ff9640 1 -- 172.21.15.1:0/931456159 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fb028053b90 con 0x7fb0380faa50 2024-06-18T02:37:49.385 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:37:49.385 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-18T02:37:49.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:49.387+0000 7fb03e844640 1 -- 172.21.15.1:0/931456159 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fb020065430 msgr2=0x7fb0200678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:37:49.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:49.387+0000 7fb03e844640 1 --2- 172.21.15.1:0/931456159 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fb020065430 0x7fb0200678e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb02c000c00 tx=0x7fb02c009da0 comp rx=0 tx=0).stop 2024-06-18T02:37:49.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:49.387+0000 7fb03e844640 1 -- 172.21.15.1:0/931456159 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0380faa50 msgr2=0x7fb038113180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:37:49.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:49.387+0000 7fb03e844640 1 --2- 172.21.15.1:0/931456159 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0380faa50 0x7fb038113180 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fb02800b790 tx=0x7fb0280091d0 comp rx=0 tx=0).stop 2024-06-18T02:37:49.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:49.387+0000 7fb03e844640 1 -- 172.21.15.1:0/931456159 shutdown_connections 2024-06-18T02:37:49.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:49.387+0000 7fb03e844640 1 --2- 172.21.15.1:0/931456159 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fb020065430 0x7fb0200678e0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:37:49.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:49.387+0000 7fb03e844640 1 --2- 172.21.15.1:0/931456159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0381023e0 0x7fb0381136c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:37:49.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:49.387+0000 7fb03e844640 1 --2- 172.21.15.1:0/931456159 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb0380faa50 0x7fb038113180 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:37:49.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:49.387+0000 7fb03e844640 1 -- 172.21.15.1:0/931456159 >> 172.21.15.1:0/931456159 conn(0x7fb038070ab0 msgr2=0x7fb0380f9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:37:49.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:49.387+0000 7fb03e844640 1 -- 172.21.15.1:0/931456159 shutdown_connections 2024-06-18T02:37:49.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:49.388+0000 7fb03e844640 1 -- 172.21.15.1:0/931456159 wait complete. 2024-06-18T02:37:50.030 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:37:49 smithi001 ceph-mon[29099]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:37:50.030 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:37:49 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/931456159' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:37:50.040 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:37:49 smithi028 ceph-mon[35823]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:37:50.040 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:37:49 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/931456159' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:37:52.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:37:51 smithi001 ceph-mon[29099]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:37:52.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:37:51 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:37:52.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:37:51 smithi028 ceph-mon[35823]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:37:52.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:37:51 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:37:53.063 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T02:37:53.341 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:37:54.127 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:37:53 smithi001 ceph-mon[29099]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:37:54.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:37:53 smithi028 ceph-mon[35823]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:37:55.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:55.055+0000 7fdeefb33640 1 -- 172.21.15.1:0/251354615 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdee8102c90 msgr2=0x7fdee80fa9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:37:55.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:55.055+0000 7fdeefb33640 1 --2- 172.21.15.1:0/251354615 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdee8102c90 0x7fdee80fa9e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fded80048d0 tx=0x7fded8009da0 comp rx=0 tx=0).stop 2024-06-18T02:37:55.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:55.055+0000 7fdeefb33640 1 -- 172.21.15.1:0/251354615 shutdown_connections 2024-06-18T02:37:55.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:55.055+0000 7fdeefb33640 1 --2- 172.21.15.1:0/251354615 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdee8102c90 0x7fdee80fa9e0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:37:55.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:55.055+0000 7fdeefb33640 1 --2- 172.21.15.1:0/251354615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdee81022f0 0x7fdee81026c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:37:55.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:55.055+0000 7fdeefb33640 1 -- 172.21.15.1:0/251354615 >> 172.21.15.1:0/251354615 conn(0x7fdee8070a40 msgr2=0x7fdee8070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:37:55.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:55.055+0000 7fdeefb33640 1 -- 172.21.15.1:0/251354615 shutdown_connections 2024-06-18T02:37:55.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:55.055+0000 7fdeefb33640 1 -- 172.21.15.1:0/251354615 wait complete. 2024-06-18T02:37:55.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:55.056+0000 7fdeefb33640 1 Processor -- start 2024-06-18T02:37:55.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:55.056+0000 7fdeefb33640 1 -- start start 2024-06-18T02:37:55.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:55.056+0000 7fdeefb33640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdee81022f0 0x7fdee81130f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:37:55.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:55.056+0000 7fdeefb33640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdee8102c90 0x7fdee8113630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:37:55.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:55.056+0000 7fdeefb33640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdee8113c30 con 0x7fdee8102c90 2024-06-18T02:37:55.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:55.056+0000 7fdeefb33640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdee810d070 con 0x7fdee81022f0 2024-06-18T02:37:55.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:55.056+0000 7fdeed0ce640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdee8102c90 0x7fdee8113630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:37:55.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:55.056+0000 7fdeed0ce640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdee8102c90 0x7fdee8113630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55940/0 (socket says 172.21.15.1:55940) 2024-06-18T02:37:55.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:55.056+0000 7fdeed0ce640 1 -- 172.21.15.1:0/2879932109 learned_addr learned my addr 172.21.15.1:0/2879932109 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:37:55.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:55.056+0000 7fdeed8cf640 1 --2- 172.21.15.1:0/2879932109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdee81022f0 0x7fdee81130f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:37:55.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:55.056+0000 7fdeed0ce640 1 -- 172.21.15.1:0/2879932109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdee81022f0 msgr2=0x7fdee81130f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:37:55.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:55.056+0000 7fdeed0ce640 1 --2- 172.21.15.1:0/2879932109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdee81022f0 0x7fdee81130f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:37:55.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:55.056+0000 7fdeed0ce640 1 -- 172.21.15.1:0/2879932109 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fded8005040 con 0x7fdee8102c90 2024-06-18T02:37:55.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:55.056+0000 7fdeed8cf640 1 --2- 172.21.15.1:0/2879932109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdee81022f0 0x7fdee81130f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-18T02:37:55.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:55.057+0000 7fdeed0ce640 1 --2- 172.21.15.1:0/2879932109 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdee8102c90 0x7fdee8113630 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fded8000c00 tx=0x7fded8009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:37:55.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:55.057+0000 7fdede7fc640 1 -- 172.21.15.1:0/2879932109 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fded8038070 con 0x7fdee8102c90 2024-06-18T02:37:55.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:55.057+0000 7fdeefb33640 1 -- 172.21.15.1:0/2879932109 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdee810d2f0 con 0x7fdee8102c90 2024-06-18T02:37:55.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:55.057+0000 7fdede7fc640 1 -- 172.21.15.1:0/2879932109 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fded8032970 con 0x7fdee8102c90 2024-06-18T02:37:55.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:55.057+0000 7fdede7fc640 1 -- 172.21.15.1:0/2879932109 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fded803d980 con 0x7fdee8102c90 2024-06-18T02:37:55.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:55.057+0000 7fdeefb33640 1 -- 172.21.15.1:0/2879932109 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdee810d7b0 con 0x7fdee8102c90 2024-06-18T02:37:55.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:55.058+0000 7fdeefb33640 1 -- 172.21.15.1:0/2879932109 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdee8065510 con 0x7fdee8102c90 2024-06-18T02:37:55.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:55.058+0000 7fdede7fc640 1 -- 172.21.15.1:0/2879932109 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7fded8030020 con 0x7fdee8102c90 2024-06-18T02:37:55.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:55.059+0000 7fdede7fc640 1 --2- 172.21.15.1:0/2879932109 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fded4065430 0x7fded40678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:37:55.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:55.059+0000 7fdede7fc640 1 -- 172.21.15.1:0/2879932109 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fded80a25f0 con 0x7fdee8102c90 2024-06-18T02:37:55.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:55.059+0000 7fdeed8cf640 1 --2- 172.21.15.1:0/2879932109 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fded4065430 0x7fded40678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:37:55.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:55.059+0000 7fdeed8cf640 1 --2- 172.21.15.1:0/2879932109 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fded4065430 0x7fded40678e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fdee8065260 tx=0x7fdee400b040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:37:55.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:55.063+0000 7fdede7fc640 1 -- 172.21.15.1:0/2879932109 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fded8070aa0 con 0x7fdee8102c90 2024-06-18T02:37:55.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:55.280+0000 7fdeefb33640 1 -- 172.21.15.1:0/2879932109 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fdee810e760 con 0x7fdee8102c90 2024-06-18T02:37:55.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:55.280+0000 7fdede7fc640 1 -- 172.21.15.1:0/2879932109 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fded80740c0 con 0x7fdee8102c90 2024-06-18T02:37:55.280 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:37:55.280 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-18T02:37:55.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:55.282+0000 7fdeefb33640 1 -- 172.21.15.1:0/2879932109 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fded4065430 msgr2=0x7fded40678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:37:55.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:55.282+0000 7fdeefb33640 1 --2- 172.21.15.1:0/2879932109 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fded4065430 0x7fded40678e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fdee8065260 tx=0x7fdee400b040 comp rx=0 tx=0).stop 2024-06-18T02:37:55.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:55.282+0000 7fdeefb33640 1 -- 172.21.15.1:0/2879932109 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdee8102c90 msgr2=0x7fdee8113630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:37:55.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:55.282+0000 7fdeefb33640 1 --2- 172.21.15.1:0/2879932109 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdee8102c90 0x7fdee8113630 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fded8000c00 tx=0x7fded8009da0 comp rx=0 tx=0).stop 2024-06-18T02:37:55.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:55.282+0000 7fdeefb33640 1 -- 172.21.15.1:0/2879932109 shutdown_connections 2024-06-18T02:37:55.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:55.282+0000 7fdeefb33640 1 --2- 172.21.15.1:0/2879932109 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fded4065430 0x7fded40678e0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:37:55.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:55.282+0000 7fdeefb33640 1 --2- 172.21.15.1:0/2879932109 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdee8102c90 0x7fdee8113630 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:37:55.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:55.282+0000 7fdeefb33640 1 --2- 172.21.15.1:0/2879932109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdee81022f0 0x7fdee81130f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:37:55.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:55.282+0000 7fdeefb33640 1 -- 172.21.15.1:0/2879932109 >> 172.21.15.1:0/2879932109 conn(0x7fdee8070a40 msgr2=0x7fdee81050b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:37:55.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:55.282+0000 7fdeefb33640 1 -- 172.21.15.1:0/2879932109 shutdown_connections 2024-06-18T02:37:55.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:37:55.282+0000 7fdeefb33640 1 -- 172.21.15.1:0/2879932109 wait complete. 2024-06-18T02:37:56.127 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:37:55 smithi001 ceph-mon[29099]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:37:56.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:37:55 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/2879932109' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:37:56.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:37:55 smithi028 ceph-mon[35823]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:37:56.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:37:55 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/2879932109' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:37:58.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:37:57 smithi001 ceph-mon[29099]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:37:58.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:37:57 smithi028 ceph-mon[35823]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:37:58.975 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T02:37:59.250 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:38:00.104 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:37:59 smithi001 ceph-mon[29099]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:38:00.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:37:59 smithi028 ceph-mon[35823]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:38:00.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:00.933+0000 7f3ed819f640 1 -- 172.21.15.1:0/3690072888 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ed0102c40 msgr2=0x7f3ed00fc2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:38:00.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:00.933+0000 7f3ed819f640 1 --2- 172.21.15.1:0/3690072888 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ed0102c40 0x7f3ed00fc2d0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f3ec40048d0 tx=0x7f3ec4009da0 comp rx=0 tx=0).stop 2024-06-18T02:38:00.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:00.933+0000 7f3ed819f640 1 -- 172.21.15.1:0/3690072888 shutdown_connections 2024-06-18T02:38:00.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:00.933+0000 7f3ed819f640 1 --2- 172.21.15.1:0/3690072888 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ed0102c40 0x7f3ed00fc2d0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:38:00.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:00.933+0000 7f3ed819f640 1 --2- 172.21.15.1:0/3690072888 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ed01022a0 0x7f3ed0102670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:38:00.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:00.933+0000 7f3ed819f640 1 -- 172.21.15.1:0/3690072888 >> 172.21.15.1:0/3690072888 conn(0x7f3ed00f7e50 msgr2=0x7f3ed00fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:38:00.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:00.933+0000 7f3ed819f640 1 -- 172.21.15.1:0/3690072888 shutdown_connections 2024-06-18T02:38:00.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:00.933+0000 7f3ed819f640 1 -- 172.21.15.1:0/3690072888 wait complete. 2024-06-18T02:38:00.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:00.933+0000 7f3ed819f640 1 Processor -- start 2024-06-18T02:38:00.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:00.933+0000 7f3ed819f640 1 -- start start 2024-06-18T02:38:00.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:00.934+0000 7f3ed819f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ed01022a0 0x7f3ed01130b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:38:00.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:00.934+0000 7f3ed819f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ed0102c40 0x7f3ed01135f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:38:00.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:00.934+0000 7f3ed819f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed0113c80 con 0x7f3ed01022a0 2024-06-18T02:38:00.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:00.934+0000 7f3ed819f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed010d030 con 0x7f3ed0102c40 2024-06-18T02:38:00.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:00.934+0000 7f3ed5f3b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ed01022a0 0x7f3ed01130b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:38:00.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:00.934+0000 7f3ed5f3b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ed01022a0 0x7f3ed01130b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55970/0 (socket says 172.21.15.1:55970) 2024-06-18T02:38:00.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:00.934+0000 7f3ed5f3b640 1 -- 172.21.15.1:0/2038361671 learned_addr learned my addr 172.21.15.1:0/2038361671 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:38:00.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:00.934+0000 7f3ed573a640 1 --2- 172.21.15.1:0/2038361671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ed0102c40 0x7f3ed01135f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:38:00.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:00.934+0000 7f3ed5f3b640 1 -- 172.21.15.1:0/2038361671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ed0102c40 msgr2=0x7f3ed01135f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:38:00.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:00.934+0000 7f3ed5f3b640 1 --2- 172.21.15.1:0/2038361671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ed0102c40 0x7f3ed01135f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:38:00.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:00.934+0000 7f3ed5f3b640 1 -- 172.21.15.1:0/2038361671 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ec4005040 con 0x7f3ed01022a0 2024-06-18T02:38:00.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:00.934+0000 7f3ed573a640 1 --2- 172.21.15.1:0/2038361671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ed0102c40 0x7f3ed01135f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-18T02:38:00.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:00.934+0000 7f3ed5f3b640 1 --2- 172.21.15.1:0/2038361671 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ed01022a0 0x7f3ed01130b0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f3ec000bbb0 tx=0x7f3ec0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:38:00.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:00.934+0000 7f3ece7fc640 1 -- 172.21.15.1:0/2038361671 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ec000c9c0 con 0x7f3ed01022a0 2024-06-18T02:38:00.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:00.934+0000 7f3ed819f640 1 -- 172.21.15.1:0/2038361671 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ed010d310 con 0x7f3ed01022a0 2024-06-18T02:38:00.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:00.934+0000 7f3ece7fc640 1 -- 172.21.15.1:0/2038361671 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3ec00092e0 con 0x7f3ed01022a0 2024-06-18T02:38:00.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:00.934+0000 7f3ece7fc640 1 -- 172.21.15.1:0/2038361671 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ec0007640 con 0x7f3ed01022a0 2024-06-18T02:38:00.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:00.935+0000 7f3ed819f640 1 -- 172.21.15.1:0/2038361671 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ed010d830 con 0x7f3ed01022a0 2024-06-18T02:38:00.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:00.936+0000 7f3ed819f640 1 -- 172.21.15.1:0/2038361671 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ed00fda10 con 0x7f3ed01022a0 2024-06-18T02:38:00.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:00.936+0000 7f3ece7fc640 1 -- 172.21.15.1:0/2038361671 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7f3ec0012070 con 0x7f3ed01022a0 2024-06-18T02:38:00.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:00.936+0000 7f3ece7fc640 1 --2- 172.21.15.1:0/2038361671 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f3eb8065430 0x7f3eb80678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:38:00.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:00.936+0000 7f3ece7fc640 1 -- 172.21.15.1:0/2038361671 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f3ec0082570 con 0x7f3ed01022a0 2024-06-18T02:38:00.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:00.936+0000 7f3ed573a640 1 --2- 172.21.15.1:0/2038361671 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f3eb8065430 0x7f3eb80678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:38:00.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:00.937+0000 7f3ed573a640 1 --2- 172.21.15.1:0/2038361671 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f3eb8065430 0x7f3eb80678e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3ed010e970 tx=0x7f3ec4035040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:38:00.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:00.941+0000 7f3ece7fc640 1 -- 172.21.15.1:0/2038361671 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f3ec0050a20 con 0x7f3ed01022a0 2024-06-18T02:38:01.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:01.152+0000 7f3ed819f640 1 -- 172.21.15.1:0/2038361671 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f3ed004b4b0 con 0x7f3ed01022a0 2024-06-18T02:38:01.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:01.152+0000 7f3ece7fc640 1 -- 172.21.15.1:0/2038361671 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f3ec0054040 con 0x7f3ed01022a0 2024-06-18T02:38:01.155 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:38:01.155 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-18T02:38:01.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:01.154+0000 7f3ed819f640 1 -- 172.21.15.1:0/2038361671 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f3eb8065430 msgr2=0x7f3eb80678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:38:01.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:01.154+0000 7f3ed819f640 1 --2- 172.21.15.1:0/2038361671 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f3eb8065430 0x7f3eb80678e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3ed010e970 tx=0x7f3ec4035040 comp rx=0 tx=0).stop 2024-06-18T02:38:01.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:01.154+0000 7f3ed819f640 1 -- 172.21.15.1:0/2038361671 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ed01022a0 msgr2=0x7f3ed01130b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:38:01.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:01.154+0000 7f3ed819f640 1 --2- 172.21.15.1:0/2038361671 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ed01022a0 0x7f3ed01130b0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f3ec000bbb0 tx=0x7f3ec0009840 comp rx=0 tx=0).stop 2024-06-18T02:38:01.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:01.155+0000 7f3ed819f640 1 -- 172.21.15.1:0/2038361671 shutdown_connections 2024-06-18T02:38:01.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:01.155+0000 7f3ed819f640 1 --2- 172.21.15.1:0/2038361671 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f3eb8065430 0x7f3eb80678e0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:38:01.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:01.155+0000 7f3ed819f640 1 --2- 172.21.15.1:0/2038361671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ed0102c40 0x7f3ed01135f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:38:01.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:01.155+0000 7f3ed819f640 1 --2- 172.21.15.1:0/2038361671 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ed01022a0 0x7f3ed01130b0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:38:01.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:01.155+0000 7f3ed819f640 1 -- 172.21.15.1:0/2038361671 >> 172.21.15.1:0/2038361671 conn(0x7f3ed00f7e50 msgr2=0x7f3ed00f8d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:38:01.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:01.155+0000 7f3ed819f640 1 -- 172.21.15.1:0/2038361671 shutdown_connections 2024-06-18T02:38:01.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:01.155+0000 7f3ed819f640 1 -- 172.21.15.1:0/2038361671 wait complete. 2024-06-18T02:38:02.127 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:38:01 smithi001 ceph-mon[29099]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:38:02.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:38:01 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/2038361671' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:38:02.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:38:01 smithi028 ceph-mon[35823]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:38:02.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:38:01 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/2038361671' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:38:03.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:38:03 smithi001 ceph-mon[29099]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:38:04.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:38:03 smithi028 ceph-mon[35823]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:38:04.910 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T02:38:05.190 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:38:05.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:38:05 smithi001 ceph-mon[29099]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:38:06.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:38:05 smithi028 ceph-mon[35823]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:38:06.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:06.974+0000 7fe70bb1d640 1 -- 172.21.15.1:0/1399769513 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe704102c90 msgr2=0x7fe7040fc320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:38:06.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:06.974+0000 7fe70bb1d640 1 --2- 172.21.15.1:0/1399769513 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe704102c90 0x7fe7040fc320 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fe6f40048d0 tx=0x7fe6f4009da0 comp rx=0 tx=0).stop 2024-06-18T02:38:06.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:06.974+0000 7fe70bb1d640 1 -- 172.21.15.1:0/1399769513 shutdown_connections 2024-06-18T02:38:06.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:06.974+0000 7fe70bb1d640 1 --2- 172.21.15.1:0/1399769513 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe704102c90 0x7fe7040fc320 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:38:06.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:06.974+0000 7fe70bb1d640 1 --2- 172.21.15.1:0/1399769513 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7041022f0 0x7fe7041026c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:38:06.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:06.974+0000 7fe70bb1d640 1 -- 172.21.15.1:0/1399769513 >> 172.21.15.1:0/1399769513 conn(0x7fe7040f7f00 msgr2=0x7fe7040fa310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:38:06.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:06.974+0000 7fe70bb1d640 1 -- 172.21.15.1:0/1399769513 shutdown_connections 2024-06-18T02:38:06.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:06.974+0000 7fe70bb1d640 1 -- 172.21.15.1:0/1399769513 wait complete. 2024-06-18T02:38:06.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:06.975+0000 7fe70bb1d640 1 Processor -- start 2024-06-18T02:38:06.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:06.975+0000 7fe70bb1d640 1 -- start start 2024-06-18T02:38:06.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:06.975+0000 7fe70bb1d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7041022f0 0x7fe70406f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:38:06.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:06.975+0000 7fe70bb1d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe704102c90 0x7fe70406fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:38:06.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:06.975+0000 7fe70bb1d640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7040702d0 con 0x7fe7041022f0 2024-06-18T02:38:06.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:06.975+0000 7fe70bb1d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe704070440 con 0x7fe704102c90 2024-06-18T02:38:06.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:06.975+0000 7fe7098b9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7041022f0 0x7fe70406f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:38:06.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:06.975+0000 7fe7098b9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7041022f0 0x7fe70406f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59382/0 (socket says 172.21.15.1:59382) 2024-06-18T02:38:06.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:06.975+0000 7fe7098b9640 1 -- 172.21.15.1:0/2916152041 learned_addr learned my addr 172.21.15.1:0/2916152041 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:38:06.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:06.975+0000 7fe7098b9640 1 -- 172.21.15.1:0/2916152041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe704102c90 msgr2=0x7fe70406fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:38:06.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:06.975+0000 7fe7098b9640 1 --2- 172.21.15.1:0/2916152041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe704102c90 0x7fe70406fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:38:06.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:06.975+0000 7fe7098b9640 1 -- 172.21.15.1:0/2916152041 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6f4005040 con 0x7fe7041022f0 2024-06-18T02:38:06.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:06.975+0000 7fe7098b9640 1 --2- 172.21.15.1:0/2916152041 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7041022f0 0x7fe70406f770 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fe70000b790 tx=0x7fe7000091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:38:06.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:06.976+0000 7fe6fa7fc640 1 -- 172.21.15.1:0/2916152041 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe70000c700 con 0x7fe7041022f0 2024-06-18T02:38:06.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:06.976+0000 7fe6fa7fc640 1 -- 172.21.15.1:0/2916152041 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe70000cd30 con 0x7fe7041022f0 2024-06-18T02:38:06.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:06.976+0000 7fe70bb1d640 1 -- 172.21.15.1:0/2916152041 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe704119360 con 0x7fe7041022f0 2024-06-18T02:38:06.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:06.976+0000 7fe6fa7fc640 1 -- 172.21.15.1:0/2916152041 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe700013370 con 0x7fe7041022f0 2024-06-18T02:38:06.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:06.976+0000 7fe70bb1d640 1 -- 172.21.15.1:0/2916152041 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe704119820 con 0x7fe7041022f0 2024-06-18T02:38:06.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:06.977+0000 7fe70bb1d640 1 -- 172.21.15.1:0/2916152041 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7040fda60 con 0x7fe7041022f0 2024-06-18T02:38:06.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:06.977+0000 7fe6fa7fc640 1 -- 172.21.15.1:0/2916152041 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7fe700012070 con 0x7fe7041022f0 2024-06-18T02:38:06.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:06.978+0000 7fe6fa7fc640 1 --2- 172.21.15.1:0/2916152041 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fe6f0065430 0x7fe6f00678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:38:06.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:06.978+0000 7fe6fa7fc640 1 -- 172.21.15.1:0/2916152041 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fe700081fd0 con 0x7fe7041022f0 2024-06-18T02:38:06.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:06.978+0000 7fe7090b8640 1 --2- 172.21.15.1:0/2916152041 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fe6f0065430 0x7fe6f00678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:38:06.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:06.978+0000 7fe7090b8640 1 --2- 172.21.15.1:0/2916152041 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fe6f0065430 0x7fe6f00678e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fe6f402a530 tx=0x7fe6f4009da0 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:38:06.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:06.982+0000 7fe6fa7fc640 1 -- 172.21.15.1:0/2916152041 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fe700050500 con 0x7fe7041022f0 2024-06-18T02:38:07.127 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:38:06 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:38:07.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:07.190+0000 7fe70bb1d640 1 -- 172.21.15.1:0/2916152041 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe7040fbb00 con 0x7fe7041022f0 2024-06-18T02:38:07.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:07.191+0000 7fe6fa7fc640 1 -- 172.21.15.1:0/2916152041 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fe700053b20 con 0x7fe7041022f0 2024-06-18T02:38:07.191 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:38:07.191 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-18T02:38:07.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:07.193+0000 7fe70bb1d640 1 -- 172.21.15.1:0/2916152041 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fe6f0065430 msgr2=0x7fe6f00678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:38:07.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:07.193+0000 7fe70bb1d640 1 --2- 172.21.15.1:0/2916152041 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fe6f0065430 0x7fe6f00678e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fe6f402a530 tx=0x7fe6f4009da0 comp rx=0 tx=0).stop 2024-06-18T02:38:07.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:07.193+0000 7fe70bb1d640 1 -- 172.21.15.1:0/2916152041 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7041022f0 msgr2=0x7fe70406f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:38:07.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:07.193+0000 7fe70bb1d640 1 --2- 172.21.15.1:0/2916152041 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7041022f0 0x7fe70406f770 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fe70000b790 tx=0x7fe7000091d0 comp rx=0 tx=0).stop 2024-06-18T02:38:07.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:07.193+0000 7fe70bb1d640 1 -- 172.21.15.1:0/2916152041 shutdown_connections 2024-06-18T02:38:07.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:07.193+0000 7fe70bb1d640 1 --2- 172.21.15.1:0/2916152041 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fe6f0065430 0x7fe6f00678e0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:38:07.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:07.193+0000 7fe70bb1d640 1 --2- 172.21.15.1:0/2916152041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe704102c90 0x7fe70406fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:38:07.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:07.193+0000 7fe70bb1d640 1 --2- 172.21.15.1:0/2916152041 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7041022f0 0x7fe70406f770 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:38:07.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:07.193+0000 7fe70bb1d640 1 -- 172.21.15.1:0/2916152041 >> 172.21.15.1:0/2916152041 conn(0x7fe7040f7f00 msgr2=0x7fe7040f8e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:38:07.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:07.193+0000 7fe70bb1d640 1 -- 172.21.15.1:0/2916152041 shutdown_connections 2024-06-18T02:38:07.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:07.193+0000 7fe70bb1d640 1 -- 172.21.15.1:0/2916152041 wait complete. 2024-06-18T02:38:07.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:38:06 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:38:07.872 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:38:07 smithi001 ceph-mon[29099]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:38:07.872 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:38:07 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/2916152041' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:38:08.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:38:07 smithi028 ceph-mon[35823]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:38:08.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:38:07 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/2916152041' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:38:10.127 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:38:09 smithi001 ceph-mon[29099]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:38:10.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:38:09 smithi028 ceph-mon[35823]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:38:10.900 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T02:38:11.176 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:38:11.870 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:38:11 smithi001 ceph-mon[29099]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:38:12.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:38:11 smithi028 ceph-mon[35823]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:38:12.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:12.869+0000 7f08457a2640 1 -- 172.21.15.1:0/519187824 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0840102ae0 msgr2=0x7f08400fa830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:38:12.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:12.869+0000 7f08457a2640 1 --2- 172.21.15.1:0/519187824 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0840102ae0 0x7f08400fa830 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f08340048d0 tx=0x7f0834009da0 comp rx=0 tx=0).stop 2024-06-18T02:38:12.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:12.869+0000 7f08457a2640 1 -- 172.21.15.1:0/519187824 shutdown_connections 2024-06-18T02:38:12.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:12.869+0000 7f08457a2640 1 --2- 172.21.15.1:0/519187824 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0840102ae0 0x7f08400fa830 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:38:12.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:12.869+0000 7f08457a2640 1 --2- 172.21.15.1:0/519187824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0840102140 0x7f0840102510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:38:12.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:12.869+0000 7f08457a2640 1 -- 172.21.15.1:0/519187824 >> 172.21.15.1:0/519187824 conn(0x7f0840070890 msgr2=0x7f0840070c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:38:12.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:12.869+0000 7f08457a2640 1 -- 172.21.15.1:0/519187824 shutdown_connections 2024-06-18T02:38:12.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:12.869+0000 7f08457a2640 1 -- 172.21.15.1:0/519187824 wait complete. 2024-06-18T02:38:12.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:12.870+0000 7f08457a2640 1 Processor -- start 2024-06-18T02:38:12.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:12.870+0000 7f08457a2640 1 -- start start 2024-06-18T02:38:12.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:12.870+0000 7f08457a2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0840102140 0x7f0840113170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:38:12.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:12.870+0000 7f08457a2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0840102ae0 0x7f08401136b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:38:12.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:12.870+0000 7f08457a2640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0840113d40 con 0x7f0840102ae0 2024-06-18T02:38:12.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:12.870+0000 7f08457a2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f084010d0f0 con 0x7f0840102140 2024-06-18T02:38:12.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:12.870+0000 7f083f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0840102ae0 0x7f08401136b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:38:12.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:12.870+0000 7f083f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0840102ae0 0x7f08401136b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59400/0 (socket says 172.21.15.1:59400) 2024-06-18T02:38:12.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:12.870+0000 7f083f7fe640 1 -- 172.21.15.1:0/3745256484 learned_addr learned my addr 172.21.15.1:0/3745256484 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:38:12.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:12.871+0000 7f083f7fe640 1 -- 172.21.15.1:0/3745256484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0840102140 msgr2=0x7f0840113170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:38:12.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:12.871+0000 7f083f7fe640 1 --2- 172.21.15.1:0/3745256484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0840102140 0x7f0840113170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:38:12.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:12.871+0000 7f083f7fe640 1 -- 172.21.15.1:0/3745256484 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0834005040 con 0x7f0840102ae0 2024-06-18T02:38:12.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:12.871+0000 7f083f7fe640 1 --2- 172.21.15.1:0/3745256484 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0840102ae0 0x7f08401136b0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f083402a530 tx=0x7f0834009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:38:12.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:12.871+0000 7f083cff9640 1 -- 172.21.15.1:0/3745256484 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0834038070 con 0x7f0840102ae0 2024-06-18T02:38:12.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:12.871+0000 7f083cff9640 1 -- 172.21.15.1:0/3745256484 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0834008ee0 con 0x7f0840102ae0 2024-06-18T02:38:12.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:12.871+0000 7f08457a2640 1 -- 172.21.15.1:0/3745256484 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f084010d370 con 0x7f0840102ae0 2024-06-18T02:38:12.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:12.871+0000 7f083cff9640 1 -- 172.21.15.1:0/3745256484 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0834033730 con 0x7f0840102ae0 2024-06-18T02:38:12.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:12.871+0000 7f08457a2640 1 -- 172.21.15.1:0/3745256484 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f084010d830 con 0x7f0840102ae0 2024-06-18T02:38:12.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:12.872+0000 7f08457a2640 1 -- 172.21.15.1:0/3745256484 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0840065350 con 0x7f0840102ae0 2024-06-18T02:38:12.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:12.873+0000 7f083cff9640 1 -- 172.21.15.1:0/3745256484 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7f0834030020 con 0x7f0840102ae0 2024-06-18T02:38:12.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:12.873+0000 7f083cff9640 1 --2- 172.21.15.1:0/3745256484 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f0828065430 0x7f08280678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:38:12.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:12.873+0000 7f083cff9640 1 -- 172.21.15.1:0/3745256484 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f08340a21d0 con 0x7f0840102ae0 2024-06-18T02:38:12.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:12.873+0000 7f083ffff640 1 --2- 172.21.15.1:0/3745256484 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f0828065430 0x7f08280678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:38:12.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:12.875+0000 7f083ffff640 1 --2- 172.21.15.1:0/3745256484 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f0828065430 0x7f08280678e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f08300097e0 tx=0x7f0830006d20 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:38:12.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:12.878+0000 7f083cff9640 1 -- 172.21.15.1:0/3745256484 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f0834070700 con 0x7f0840102ae0 2024-06-18T02:38:13.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:13.087+0000 7f08457a2640 1 -- 172.21.15.1:0/3745256484 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f0840109380 con 0x7f0840102ae0 2024-06-18T02:38:13.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:13.087+0000 7f083cff9640 1 -- 172.21.15.1:0/3745256484 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f0834073d20 con 0x7f0840102ae0 2024-06-18T02:38:13.087 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:38:13.087 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-18T02:38:13.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:13.089+0000 7f08457a2640 1 -- 172.21.15.1:0/3745256484 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f0828065430 msgr2=0x7f08280678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:38:13.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:13.089+0000 7f08457a2640 1 --2- 172.21.15.1:0/3745256484 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f0828065430 0x7f08280678e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f08300097e0 tx=0x7f0830006d20 comp rx=0 tx=0).stop 2024-06-18T02:38:13.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:13.089+0000 7f08457a2640 1 -- 172.21.15.1:0/3745256484 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0840102ae0 msgr2=0x7f08401136b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:38:13.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:13.089+0000 7f08457a2640 1 --2- 172.21.15.1:0/3745256484 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0840102ae0 0x7f08401136b0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f083402a530 tx=0x7f0834009da0 comp rx=0 tx=0).stop 2024-06-18T02:38:13.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:13.089+0000 7f08457a2640 1 -- 172.21.15.1:0/3745256484 shutdown_connections 2024-06-18T02:38:13.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:13.090+0000 7f08457a2640 1 --2- 172.21.15.1:0/3745256484 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f0828065430 0x7f08280678e0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:38:13.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:13.090+0000 7f08457a2640 1 --2- 172.21.15.1:0/3745256484 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0840102ae0 0x7f08401136b0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:38:13.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:13.090+0000 7f08457a2640 1 --2- 172.21.15.1:0/3745256484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0840102140 0x7f0840113170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:38:13.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:13.090+0000 7f08457a2640 1 -- 172.21.15.1:0/3745256484 >> 172.21.15.1:0/3745256484 conn(0x7f0840070890 msgr2=0x7f0840104f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:38:13.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:13.090+0000 7f08457a2640 1 -- 172.21.15.1:0/3745256484 shutdown_connections 2024-06-18T02:38:13.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:13.090+0000 7f08457a2640 1 -- 172.21.15.1:0/3745256484 wait complete. 2024-06-18T02:38:13.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:38:13 smithi001 ceph-mon[29099]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:38:13.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:38:13 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/3745256484' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:38:14.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:38:13 smithi028 ceph-mon[35823]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:38:14.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:38:13 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/3745256484' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:38:16.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:38:15 smithi001 ceph-mon[29099]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:38:16.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:38:15 smithi028 ceph-mon[35823]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:38:16.732 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T02:38:17.010 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:38:18.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:38:17 smithi001 ceph-mon[29099]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:38:18.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:38:17 smithi028 ceph-mon[35823]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:38:18.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:18.602+0000 7fcf455ef640 1 -- 172.21.15.1:0/610275894 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf401022f0 msgr2=0x7fcf401026c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:38:18.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:18.602+0000 7fcf455ef640 1 --2- 172.21.15.1:0/610275894 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf401022f0 0x7fcf401026c0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fcf300048d0 tx=0x7fcf30009da0 comp rx=0 tx=0).stop 2024-06-18T02:38:18.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:18.602+0000 7fcf455ef640 1 -- 172.21.15.1:0/610275894 shutdown_connections 2024-06-18T02:38:18.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:18.602+0000 7fcf455ef640 1 --2- 172.21.15.1:0/610275894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf40102c90 0x7fcf400fc320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:38:18.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:18.602+0000 7fcf455ef640 1 --2- 172.21.15.1:0/610275894 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf401022f0 0x7fcf401026c0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:38:18.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:18.602+0000 7fcf455ef640 1 -- 172.21.15.1:0/610275894 >> 172.21.15.1:0/610275894 conn(0x7fcf400f7f00 msgr2=0x7fcf400fa310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:38:18.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:18.602+0000 7fcf455ef640 1 -- 172.21.15.1:0/610275894 shutdown_connections 2024-06-18T02:38:18.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:18.602+0000 7fcf455ef640 1 -- 172.21.15.1:0/610275894 wait complete. 2024-06-18T02:38:18.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:18.602+0000 7fcf455ef640 1 Processor -- start 2024-06-18T02:38:18.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:18.603+0000 7fcf455ef640 1 -- start start 2024-06-18T02:38:18.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:18.603+0000 7fcf455ef640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf401022f0 0x7fcf40113040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:38:18.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:18.603+0000 7fcf455ef640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf40102c90 0x7fcf40113580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:38:18.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:18.603+0000 7fcf455ef640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf40113c10 con 0x7fcf401022f0 2024-06-18T02:38:18.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:18.603+0000 7fcf455ef640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf4010d000 con 0x7fcf40102c90 2024-06-18T02:38:18.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:18.603+0000 7fcf3effd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf401022f0 0x7fcf40113040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:38:18.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:18.603+0000 7fcf3effd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf401022f0 0x7fcf40113040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44962/0 (socket says 172.21.15.1:44962) 2024-06-18T02:38:18.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:18.603+0000 7fcf3effd640 1 -- 172.21.15.1:0/482098689 learned_addr learned my addr 172.21.15.1:0/482098689 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:38:18.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:18.603+0000 7fcf367fc640 1 --2- 172.21.15.1:0/482098689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf40102c90 0x7fcf40113580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:38:18.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:18.603+0000 7fcf3effd640 1 -- 172.21.15.1:0/482098689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf40102c90 msgr2=0x7fcf40113580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:38:18.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:18.603+0000 7fcf3effd640 1 --2- 172.21.15.1:0/482098689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf40102c90 0x7fcf40113580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:38:18.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:18.603+0000 7fcf3effd640 1 -- 172.21.15.1:0/482098689 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf30005040 con 0x7fcf401022f0 2024-06-18T02:38:18.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:18.604+0000 7fcf3effd640 1 --2- 172.21.15.1:0/482098689 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf401022f0 0x7fcf40113040 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fcf30003080 tx=0x7fcf30009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:38:18.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:18.604+0000 7fcf37fff640 1 -- 172.21.15.1:0/482098689 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcf30038070 con 0x7fcf401022f0 2024-06-18T02:38:18.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:18.604+0000 7fcf455ef640 1 -- 172.21.15.1:0/482098689 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf4010d280 con 0x7fcf401022f0 2024-06-18T02:38:18.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:18.604+0000 7fcf37fff640 1 -- 172.21.15.1:0/482098689 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcf30008ee0 con 0x7fcf401022f0 2024-06-18T02:38:18.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:18.604+0000 7fcf37fff640 1 -- 172.21.15.1:0/482098689 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcf3003d870 con 0x7fcf401022f0 2024-06-18T02:38:18.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:18.604+0000 7fcf455ef640 1 -- 172.21.15.1:0/482098689 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf4010d740 con 0x7fcf401022f0 2024-06-18T02:38:18.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:18.605+0000 7fcf455ef640 1 -- 172.21.15.1:0/482098689 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf400fda60 con 0x7fcf401022f0 2024-06-18T02:38:18.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:18.605+0000 7fcf37fff640 1 -- 172.21.15.1:0/482098689 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7fcf30030020 con 0x7fcf401022f0 2024-06-18T02:38:18.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:18.606+0000 7fcf37fff640 1 --2- 172.21.15.1:0/482098689 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fcf2c065430 0x7fcf2c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:38:18.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:18.606+0000 7fcf37fff640 1 -- 172.21.15.1:0/482098689 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fcf300a2240 con 0x7fcf401022f0 2024-06-18T02:38:18.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:18.606+0000 7fcf367fc640 1 --2- 172.21.15.1:0/482098689 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fcf2c065430 0x7fcf2c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:38:18.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:18.606+0000 7fcf367fc640 1 --2- 172.21.15.1:0/482098689 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fcf2c065430 0x7fcf2c0678e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fcf4010e940 tx=0x7fcf28008040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:38:18.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:18.610+0000 7fcf37fff640 1 -- 172.21.15.1:0/482098689 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fcf30070770 con 0x7fcf401022f0 2024-06-18T02:38:18.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:18.820+0000 7fcf455ef640 1 -- 172.21.15.1:0/482098689 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fcf400fbb00 con 0x7fcf401022f0 2024-06-18T02:38:18.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:18.820+0000 7fcf37fff640 1 -- 172.21.15.1:0/482098689 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fcf30073d90 con 0x7fcf401022f0 2024-06-18T02:38:18.821 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:38:18.821 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-18T02:38:18.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:18.823+0000 7fcf455ef640 1 -- 172.21.15.1:0/482098689 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fcf2c065430 msgr2=0x7fcf2c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:38:18.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:18.823+0000 7fcf455ef640 1 --2- 172.21.15.1:0/482098689 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fcf2c065430 0x7fcf2c0678e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fcf4010e940 tx=0x7fcf28008040 comp rx=0 tx=0).stop 2024-06-18T02:38:18.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:18.823+0000 7fcf455ef640 1 -- 172.21.15.1:0/482098689 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf401022f0 msgr2=0x7fcf40113040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:38:18.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:18.823+0000 7fcf455ef640 1 --2- 172.21.15.1:0/482098689 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf401022f0 0x7fcf40113040 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fcf30003080 tx=0x7fcf30009da0 comp rx=0 tx=0).stop 2024-06-18T02:38:18.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:18.823+0000 7fcf455ef640 1 -- 172.21.15.1:0/482098689 shutdown_connections 2024-06-18T02:38:18.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:18.823+0000 7fcf455ef640 1 --2- 172.21.15.1:0/482098689 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fcf2c065430 0x7fcf2c0678e0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:38:18.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:18.823+0000 7fcf455ef640 1 --2- 172.21.15.1:0/482098689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf40102c90 0x7fcf40113580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:38:18.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:18.823+0000 7fcf455ef640 1 --2- 172.21.15.1:0/482098689 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf401022f0 0x7fcf40113040 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:38:18.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:18.823+0000 7fcf455ef640 1 -- 172.21.15.1:0/482098689 >> 172.21.15.1:0/482098689 conn(0x7fcf400f7f00 msgr2=0x7fcf400f8e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:38:18.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:18.823+0000 7fcf455ef640 1 -- 172.21.15.1:0/482098689 shutdown_connections 2024-06-18T02:38:18.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:18.823+0000 7fcf455ef640 1 -- 172.21.15.1:0/482098689 wait complete. 2024-06-18T02:38:20.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:38:19 smithi001 ceph-mon[29099]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:38:20.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:38:19 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/482098689' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:38:20.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:38:19 smithi028 ceph-mon[35823]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:38:20.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:38:19 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/482098689' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:38:22.129 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:38:21 smithi001 ceph-mon[29099]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:38:22.130 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:38:21 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:38:22.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:38:21 smithi028 ceph-mon[35823]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:38:22.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:38:21 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:38:22.503 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T02:38:22.782 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:38:23.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:38:23 smithi001 ceph-mon[29099]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:38:24.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:38:23 smithi028 ceph-mon[35823]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:38:24.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:24.423+0000 7f369d7e0640 1 -- 172.21.15.1:0/2623334917 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3698065010 msgr2=0x7f36980653e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:38:24.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:24.423+0000 7f369d7e0640 1 --2- 172.21.15.1:0/2623334917 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3698065010 0x7f36980653e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f36880048d0 tx=0x7f3688009da0 comp rx=0 tx=0).stop 2024-06-18T02:38:24.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:24.423+0000 7f369d7e0640 1 -- 172.21.15.1:0/2623334917 shutdown_connections 2024-06-18T02:38:24.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:24.423+0000 7f369d7e0640 1 --2- 172.21.15.1:0/2623334917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36980fdea0 0x7f36980fe310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:38:24.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:24.423+0000 7f369d7e0640 1 --2- 172.21.15.1:0/2623334917 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3698065010 0x7f36980653e0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:38:24.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:24.423+0000 7f369d7e0640 1 -- 172.21.15.1:0/2623334917 >> 172.21.15.1:0/2623334917 conn(0x7f3698072110 msgr2=0x7f3698072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:38:24.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:24.423+0000 7f369d7e0640 1 -- 172.21.15.1:0/2623334917 shutdown_connections 2024-06-18T02:38:24.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:24.423+0000 7f369d7e0640 1 -- 172.21.15.1:0/2623334917 wait complete. 2024-06-18T02:38:24.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:24.423+0000 7f369d7e0640 1 Processor -- start 2024-06-18T02:38:24.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:24.423+0000 7f369d7e0640 1 -- start start 2024-06-18T02:38:24.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:24.424+0000 7f369d7e0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3698065010 0x7f3698112ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:38:24.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:24.424+0000 7f369d7e0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36980fdea0 0x7f3698113530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:38:24.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:24.424+0000 7f369d7e0640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3698113b30 con 0x7f3698065010 2024-06-18T02:38:24.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:24.424+0000 7f369d7e0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3698113ca0 con 0x7f36980fdea0 2024-06-18T02:38:24.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:24.424+0000 7f3696ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3698065010 0x7f3698112ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:38:24.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:24.424+0000 7f3696ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3698065010 0x7f3698112ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47334/0 (socket says 172.21.15.1:47334) 2024-06-18T02:38:24.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:24.424+0000 7f3696ffd640 1 -- 172.21.15.1:0/3595005964 learned_addr learned my addr 172.21.15.1:0/3595005964 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:38:24.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:24.424+0000 7f36967fc640 1 --2- 172.21.15.1:0/3595005964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36980fdea0 0x7f3698113530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:38:24.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:24.424+0000 7f3696ffd640 1 -- 172.21.15.1:0/3595005964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36980fdea0 msgr2=0x7f3698113530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:38:24.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:24.424+0000 7f3696ffd640 1 --2- 172.21.15.1:0/3595005964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36980fdea0 0x7f3698113530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:38:24.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:24.424+0000 7f3696ffd640 1 -- 172.21.15.1:0/3595005964 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3688005040 con 0x7f3698065010 2024-06-18T02:38:24.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:24.424+0000 7f36967fc640 1 --2- 172.21.15.1:0/3595005964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36980fdea0 0x7f3698113530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-18T02:38:24.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:24.424+0000 7f3696ffd640 1 --2- 172.21.15.1:0/3595005964 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3698065010 0x7f3698112ff0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f36880043d0 tx=0x7f3688009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:38:24.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:24.424+0000 7f367f7fe640 1 -- 172.21.15.1:0/3595005964 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3688038070 con 0x7f3698065010 2024-06-18T02:38:24.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:24.425+0000 7f367f7fe640 1 -- 172.21.15.1:0/3595005964 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3688032970 con 0x7f3698065010 2024-06-18T02:38:24.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:24.425+0000 7f369d7e0640 1 -- 172.21.15.1:0/3595005964 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f369810d0c0 con 0x7f3698065010 2024-06-18T02:38:24.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:24.425+0000 7f367f7fe640 1 -- 172.21.15.1:0/3595005964 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f368803da60 con 0x7f3698065010 2024-06-18T02:38:24.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:24.425+0000 7f369d7e0640 1 -- 172.21.15.1:0/3595005964 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f369810d600 con 0x7f3698065010 2024-06-18T02:38:24.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:24.425+0000 7f369d7e0640 1 -- 172.21.15.1:0/3595005964 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36980fda40 con 0x7f3698065010 2024-06-18T02:38:24.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:24.426+0000 7f367f7fe640 1 -- 172.21.15.1:0/3595005964 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7f3688030020 con 0x7f3698065010 2024-06-18T02:38:24.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:24.427+0000 7f367f7fe640 1 --2- 172.21.15.1:0/3595005964 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f3680065430 0x7f36800678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:38:24.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:24.427+0000 7f36967fc640 1 --2- 172.21.15.1:0/3595005964 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f3680065430 0x7f36800678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:38:24.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:24.427+0000 7f367f7fe640 1 -- 172.21.15.1:0/3595005964 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f36880a26c0 con 0x7f3698065010 2024-06-18T02:38:24.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:24.427+0000 7f36967fc640 1 --2- 172.21.15.1:0/3595005964 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f3680065430 0x7f36800678e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f369810e8f0 tx=0x7f368c00b040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:38:24.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:24.431+0000 7f367f7fe640 1 -- 172.21.15.1:0/3595005964 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f3688070b70 con 0x7f3698065010 2024-06-18T02:38:24.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:24.641+0000 7f369d7e0640 1 -- 172.21.15.1:0/3595005964 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f3698072c20 con 0x7f3698065010 2024-06-18T02:38:24.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:24.642+0000 7f367f7fe640 1 -- 172.21.15.1:0/3595005964 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f3688074190 con 0x7f3698065010 2024-06-18T02:38:24.641 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:38:24.642 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-18T02:38:24.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:24.644+0000 7f369d7e0640 1 -- 172.21.15.1:0/3595005964 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f3680065430 msgr2=0x7f36800678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:38:24.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:24.644+0000 7f369d7e0640 1 --2- 172.21.15.1:0/3595005964 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f3680065430 0x7f36800678e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f369810e8f0 tx=0x7f368c00b040 comp rx=0 tx=0).stop 2024-06-18T02:38:24.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:24.644+0000 7f369d7e0640 1 -- 172.21.15.1:0/3595005964 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3698065010 msgr2=0x7f3698112ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:38:24.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:24.644+0000 7f369d7e0640 1 --2- 172.21.15.1:0/3595005964 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3698065010 0x7f3698112ff0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f36880043d0 tx=0x7f3688009da0 comp rx=0 tx=0).stop 2024-06-18T02:38:24.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:24.644+0000 7f369d7e0640 1 -- 172.21.15.1:0/3595005964 shutdown_connections 2024-06-18T02:38:24.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:24.644+0000 7f369d7e0640 1 --2- 172.21.15.1:0/3595005964 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f3680065430 0x7f36800678e0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:38:24.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:24.644+0000 7f369d7e0640 1 --2- 172.21.15.1:0/3595005964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36980fdea0 0x7f3698113530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:38:24.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:24.644+0000 7f369d7e0640 1 --2- 172.21.15.1:0/3595005964 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3698065010 0x7f3698112ff0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:38:24.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:24.644+0000 7f369d7e0640 1 -- 172.21.15.1:0/3595005964 >> 172.21.15.1:0/3595005964 conn(0x7f3698072110 msgr2=0x7f36980f9f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:38:24.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:24.644+0000 7f369d7e0640 1 -- 172.21.15.1:0/3595005964 shutdown_connections 2024-06-18T02:38:24.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:24.644+0000 7f369d7e0640 1 -- 172.21.15.1:0/3595005964 wait complete. 2024-06-18T02:38:24.954 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:38:24 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:38:24.954 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:38:24 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/3595005964' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:38:25.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:38:24 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:38:25.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:38:24 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/3595005964' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:38:26.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:38:25 smithi001 ceph-mon[29099]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:38:26.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:38:25 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:38:26.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:38:25 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T02:38:26.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:38:25 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:38:26.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:38:25 smithi028 ceph-mon[35823]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:38:26.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:38:25 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:38:26.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:38:25 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T02:38:26.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:38:25 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:38:28.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:38:27 smithi001 ceph-mon[29099]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:38:28.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:38:27 smithi028 ceph-mon[35823]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:38:28.298 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T02:38:28.572 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:38:30.017 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:38:29 smithi001 ceph-mon[29099]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:38:30.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:30.211+0000 7f02d766f640 1 -- 172.21.15.1:0/2212703175 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02d0102370 msgr2=0x7f02d0102740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:38:30.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:30.211+0000 7f02d766f640 1 --2- 172.21.15.1:0/2212703175 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02d0102370 0x7f02d0102740 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f02cc0048d0 tx=0x7f02cc009da0 comp rx=0 tx=0).stop 2024-06-18T02:38:30.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:30.212+0000 7f02d766f640 1 -- 172.21.15.1:0/2212703175 shutdown_connections 2024-06-18T02:38:30.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:30.212+0000 7f02d766f640 1 --2- 172.21.15.1:0/2212703175 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f02d00fa9e0 0x7f02d00fae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:38:30.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:30.212+0000 7f02d766f640 1 --2- 172.21.15.1:0/2212703175 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02d0102370 0x7f02d0102740 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:38:30.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:30.212+0000 7f02d766f640 1 -- 172.21.15.1:0/2212703175 >> 172.21.15.1:0/2212703175 conn(0x7f02d0070a40 msgr2=0x7f02d0070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:38:30.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:30.212+0000 7f02d766f640 1 -- 172.21.15.1:0/2212703175 shutdown_connections 2024-06-18T02:38:30.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:30.212+0000 7f02d766f640 1 -- 172.21.15.1:0/2212703175 wait complete. 2024-06-18T02:38:30.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:30.212+0000 7f02d766f640 1 Processor -- start 2024-06-18T02:38:30.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:30.212+0000 7f02d766f640 1 -- start start 2024-06-18T02:38:30.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:30.213+0000 7f02d766f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f02d00fa9e0 0x7f02d01130d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:38:30.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:30.213+0000 7f02d766f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02d0102370 0x7f02d0113610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:38:30.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:30.213+0000 7f02d766f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d0113c10 con 0x7f02d0102370 2024-06-18T02:38:30.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:30.213+0000 7f02d766f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d010d050 con 0x7f02d00fa9e0 2024-06-18T02:38:30.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:30.213+0000 7f02d4c0a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02d0102370 0x7f02d0113610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:38:30.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:30.213+0000 7f02d4c0a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02d0102370 0x7f02d0113610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47356/0 (socket says 172.21.15.1:47356) 2024-06-18T02:38:30.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:30.213+0000 7f02d4c0a640 1 -- 172.21.15.1:0/431712999 learned_addr learned my addr 172.21.15.1:0/431712999 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:38:30.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:30.213+0000 7f02d540b640 1 --2- 172.21.15.1:0/431712999 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f02d00fa9e0 0x7f02d01130d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:38:30.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:30.213+0000 7f02d4c0a640 1 -- 172.21.15.1:0/431712999 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f02d00fa9e0 msgr2=0x7f02d01130d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:38:30.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:30.213+0000 7f02d4c0a640 1 --2- 172.21.15.1:0/431712999 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f02d00fa9e0 0x7f02d01130d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:38:30.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:30.213+0000 7f02d4c0a640 1 -- 172.21.15.1:0/431712999 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02cc005040 con 0x7f02d0102370 2024-06-18T02:38:30.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:30.213+0000 7f02d540b640 1 --2- 172.21.15.1:0/431712999 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f02d00fa9e0 0x7f02d01130d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-18T02:38:30.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:30.213+0000 7f02d4c0a640 1 --2- 172.21.15.1:0/431712999 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02d0102370 0x7f02d0113610 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f02c000c790 tx=0x7f02c00043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:38:30.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:30.213+0000 7f02c5ffb640 1 -- 172.21.15.1:0/431712999 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f02c000a820 con 0x7f02d0102370 2024-06-18T02:38:30.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:30.214+0000 7f02d766f640 1 -- 172.21.15.1:0/431712999 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02d010d330 con 0x7f02d0102370 2024-06-18T02:38:30.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:30.214+0000 7f02c5ffb640 1 -- 172.21.15.1:0/431712999 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f02c000ae50 con 0x7f02d0102370 2024-06-18T02:38:30.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:30.214+0000 7f02c5ffb640 1 -- 172.21.15.1:0/431712999 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f02c00115e0 con 0x7f02d0102370 2024-06-18T02:38:30.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:30.214+0000 7f02d766f640 1 -- 172.21.15.1:0/431712999 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02d010d850 con 0x7f02d0102370 2024-06-18T02:38:30.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:30.215+0000 7f02d766f640 1 -- 172.21.15.1:0/431712999 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02d0065530 con 0x7f02d0102370 2024-06-18T02:38:30.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:30.215+0000 7f02c5ffb640 1 -- 172.21.15.1:0/431712999 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7f02c0012400 con 0x7f02d0102370 2024-06-18T02:38:30.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:30.216+0000 7f02c5ffb640 1 --2- 172.21.15.1:0/431712999 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f02bc065430 0x7f02bc0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:38:30.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:30.216+0000 7f02c5ffb640 1 -- 172.21.15.1:0/431712999 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f02c0082680 con 0x7f02d0102370 2024-06-18T02:38:30.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:30.216+0000 7f02d540b640 1 --2- 172.21.15.1:0/431712999 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f02bc065430 0x7f02bc0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:38:30.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:30.216+0000 7f02d540b640 1 --2- 172.21.15.1:0/431712999 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f02bc065430 0x7f02bc0678e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f02cc009f80 tx=0x7f02cc035040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:38:30.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:30.220+0000 7f02c5ffb640 1 -- 172.21.15.1:0/431712999 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f02c0050bb0 con 0x7f02d0102370 2024-06-18T02:38:30.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:38:29 smithi028 ceph-mon[35823]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:38:30.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:30.432+0000 7f02d766f640 1 -- 172.21.15.1:0/431712999 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f02d01094e0 con 0x7f02d0102370 2024-06-18T02:38:30.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:30.432+0000 7f02c5ffb640 1 -- 172.21.15.1:0/431712999 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f02c00541d0 con 0x7f02d0102370 2024-06-18T02:38:30.434 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:38:30.434 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-18T02:38:30.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:30.434+0000 7f02d766f640 1 -- 172.21.15.1:0/431712999 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f02bc065430 msgr2=0x7f02bc0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:38:30.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:30.434+0000 7f02d766f640 1 --2- 172.21.15.1:0/431712999 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f02bc065430 0x7f02bc0678e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f02cc009f80 tx=0x7f02cc035040 comp rx=0 tx=0).stop 2024-06-18T02:38:30.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:30.434+0000 7f02d766f640 1 -- 172.21.15.1:0/431712999 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02d0102370 msgr2=0x7f02d0113610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:38:30.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:30.434+0000 7f02d766f640 1 --2- 172.21.15.1:0/431712999 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02d0102370 0x7f02d0113610 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f02c000c790 tx=0x7f02c00043c0 comp rx=0 tx=0).stop 2024-06-18T02:38:30.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:30.435+0000 7f02d766f640 1 -- 172.21.15.1:0/431712999 shutdown_connections 2024-06-18T02:38:30.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:30.435+0000 7f02d766f640 1 --2- 172.21.15.1:0/431712999 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f02bc065430 0x7f02bc0678e0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:38:30.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:30.435+0000 7f02d766f640 1 --2- 172.21.15.1:0/431712999 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02d0102370 0x7f02d0113610 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:38:30.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:30.435+0000 7f02d766f640 1 --2- 172.21.15.1:0/431712999 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f02d00fa9e0 0x7f02d01130d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:38:30.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:30.435+0000 7f02d766f640 1 -- 172.21.15.1:0/431712999 >> 172.21.15.1:0/431712999 conn(0x7f02d0070a40 msgr2=0x7f02d00f9c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:38:30.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:30.435+0000 7f02d766f640 1 -- 172.21.15.1:0/431712999 shutdown_connections 2024-06-18T02:38:30.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:30.435+0000 7f02d766f640 1 -- 172.21.15.1:0/431712999 wait complete. 2024-06-18T02:38:31.130 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:38:30 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/431712999' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:38:31.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:38:30 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/431712999' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:38:32.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:38:31 smithi001 ceph-mon[29099]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:38:32.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:38:31 smithi028 ceph-mon[35823]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:38:34.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:38:33 smithi001 ceph-mon[29099]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:38:34.164 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T02:38:34.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:38:33 smithi028 ceph-mon[35823]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:38:34.443 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:38:35.868 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:38:35 smithi001 ceph-mon[29099]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:38:36.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:36.134+0000 7fdc31e4e640 1 -- 172.21.15.1:0/1051454569 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc2c0fbe80 msgr2=0x7fdc2c0fcd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:38:36.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:36.134+0000 7fdc31e4e640 1 --2- 172.21.15.1:0/1051454569 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc2c0fbe80 0x7fdc2c0fcd80 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fdc200048d0 tx=0x7fdc20009da0 comp rx=0 tx=0).stop 2024-06-18T02:38:36.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:36.134+0000 7fdc31e4e640 1 -- 172.21.15.1:0/1051454569 shutdown_connections 2024-06-18T02:38:36.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:36.134+0000 7fdc31e4e640 1 --2- 172.21.15.1:0/1051454569 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc2c0fbe80 0x7fdc2c0fcd80 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:38:36.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:36.134+0000 7fdc31e4e640 1 --2- 172.21.15.1:0/1051454569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc2c0fb570 0x7fdc2c0fb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:38:36.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:36.134+0000 7fdc31e4e640 1 -- 172.21.15.1:0/1051454569 >> 172.21.15.1:0/1051454569 conn(0x7fdc2c072110 msgr2=0x7fdc2c072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:38:36.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:36.134+0000 7fdc31e4e640 1 -- 172.21.15.1:0/1051454569 shutdown_connections 2024-06-18T02:38:36.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:36.134+0000 7fdc31e4e640 1 -- 172.21.15.1:0/1051454569 wait complete. 2024-06-18T02:38:36.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:36.135+0000 7fdc31e4e640 1 Processor -- start 2024-06-18T02:38:36.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:36.135+0000 7fdc31e4e640 1 -- start start 2024-06-18T02:38:36.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:36.135+0000 7fdc31e4e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc2c0fb570 0x7fdc2c10df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:38:36.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:36.135+0000 7fdc31e4e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc2c0fbe80 0x7fdc2c10e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:38:36.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:36.135+0000 7fdc31e4e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2c10eb20 con 0x7fdc2c0fb570 2024-06-18T02:38:36.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:36.135+0000 7fdc31e4e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2c10ec90 con 0x7fdc2c0fbe80 2024-06-18T02:38:36.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:36.135+0000 7fdc2b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc2c0fb570 0x7fdc2c10df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:38:36.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:36.135+0000 7fdc2b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc2c0fb570 0x7fdc2c10df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56414/0 (socket says 172.21.15.1:56414) 2024-06-18T02:38:36.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:36.135+0000 7fdc2b7fe640 1 -- 172.21.15.1:0/4133903004 learned_addr learned my addr 172.21.15.1:0/4133903004 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:38:36.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:36.135+0000 7fdc2b7fe640 1 -- 172.21.15.1:0/4133903004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc2c0fbe80 msgr2=0x7fdc2c10e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:38:36.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:36.135+0000 7fdc2b7fe640 1 --2- 172.21.15.1:0/4133903004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc2c0fbe80 0x7fdc2c10e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:38:36.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:36.135+0000 7fdc2b7fe640 1 -- 172.21.15.1:0/4133903004 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc20005040 con 0x7fdc2c0fb570 2024-06-18T02:38:36.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:36.136+0000 7fdc2b7fe640 1 --2- 172.21.15.1:0/4133903004 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc2c0fb570 0x7fdc2c10df90 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c002a50 tx=0x7fdc1c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:38:36.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:36.136+0000 7fdc13fff640 1 -- 172.21.15.1:0/4133903004 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc1c007c30 con 0x7fdc2c0fb570 2024-06-18T02:38:36.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:36.136+0000 7fdc31e4e640 1 -- 172.21.15.1:0/4133903004 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc2c117000 con 0x7fdc2c0fb570 2024-06-18T02:38:36.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:36.136+0000 7fdc31e4e640 1 -- 172.21.15.1:0/4133903004 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc2c117510 con 0x7fdc2c0fb570 2024-06-18T02:38:36.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:36.136+0000 7fdc13fff640 1 -- 172.21.15.1:0/4133903004 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdc1c00f040 con 0x7fdc2c0fb570 2024-06-18T02:38:36.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:36.136+0000 7fdc13fff640 1 -- 172.21.15.1:0/4133903004 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc1c00db50 con 0x7fdc2c0fb570 2024-06-18T02:38:36.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:36.137+0000 7fdc31e4e640 1 -- 172.21.15.1:0/4133903004 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc18005210 con 0x7fdc2c0fb570 2024-06-18T02:38:36.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:36.137+0000 7fdc13fff640 1 -- 172.21.15.1:0/4133903004 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7fdc1c009730 con 0x7fdc2c0fb570 2024-06-18T02:38:36.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:36.138+0000 7fdc13fff640 1 --2- 172.21.15.1:0/4133903004 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fdc14065430 0x7fdc140678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:38:36.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:36.138+0000 7fdc13fff640 1 -- 172.21.15.1:0/4133903004 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fdc1c081b60 con 0x7fdc2c0fb570 2024-06-18T02:38:36.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:36.138+0000 7fdc2affd640 1 --2- 172.21.15.1:0/4133903004 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fdc14065430 0x7fdc140678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:38:36.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:36.138+0000 7fdc2affd640 1 --2- 172.21.15.1:0/4133903004 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fdc14065430 0x7fdc140678e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fdc20032430 tx=0x7fdc20009da0 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:38:36.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:36.143+0000 7fdc13fff640 1 -- 172.21.15.1:0/4133903004 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fdc1c04cdf0 con 0x7fdc2c0fb570 2024-06-18T02:38:36.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:38:35 smithi028 ceph-mon[35823]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:38:36.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:36.356+0000 7fdc31e4e640 1 -- 172.21.15.1:0/4133903004 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fdc18005a40 con 0x7fdc2c0fb570 2024-06-18T02:38:36.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:36.356+0000 7fdc13fff640 1 -- 172.21.15.1:0/4133903004 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fdc1c050090 con 0x7fdc2c0fb570 2024-06-18T02:38:36.357 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:38:36.357 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-18T02:38:36.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:36.358+0000 7fdc31e4e640 1 -- 172.21.15.1:0/4133903004 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fdc14065430 msgr2=0x7fdc140678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:38:36.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:36.358+0000 7fdc31e4e640 1 --2- 172.21.15.1:0/4133903004 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fdc14065430 0x7fdc140678e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fdc20032430 tx=0x7fdc20009da0 comp rx=0 tx=0).stop 2024-06-18T02:38:36.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:36.358+0000 7fdc31e4e640 1 -- 172.21.15.1:0/4133903004 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc2c0fb570 msgr2=0x7fdc2c10df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:38:36.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:36.358+0000 7fdc31e4e640 1 --2- 172.21.15.1:0/4133903004 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc2c0fb570 0x7fdc2c10df90 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c002a50 tx=0x7fdc1c004600 comp rx=0 tx=0).stop 2024-06-18T02:38:36.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:36.358+0000 7fdc31e4e640 1 -- 172.21.15.1:0/4133903004 shutdown_connections 2024-06-18T02:38:36.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:36.358+0000 7fdc31e4e640 1 --2- 172.21.15.1:0/4133903004 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fdc14065430 0x7fdc140678e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:38:36.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:36.358+0000 7fdc31e4e640 1 --2- 172.21.15.1:0/4133903004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc2c0fbe80 0x7fdc2c10e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:38:36.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:36.358+0000 7fdc31e4e640 1 --2- 172.21.15.1:0/4133903004 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdc2c0fb570 0x7fdc2c10df90 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:38:36.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:36.358+0000 7fdc31e4e640 1 -- 172.21.15.1:0/4133903004 >> 172.21.15.1:0/4133903004 conn(0x7fdc2c072110 msgr2=0x7fdc2c0ffea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:38:36.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:36.358+0000 7fdc31e4e640 1 -- 172.21.15.1:0/4133903004 shutdown_connections 2024-06-18T02:38:36.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:36.358+0000 7fdc31e4e640 1 -- 172.21.15.1:0/4133903004 wait complete. 2024-06-18T02:38:37.003 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:38:36 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/4133903004' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:38:37.003 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:38:36 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:38:37.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:38:36 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/4133903004' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:38:37.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:38:36 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:38:38.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:38:37 smithi001 ceph-mon[29099]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:38:38.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:38:37 smithi028 ceph-mon[35823]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:38:40.030 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T02:38:40.127 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:38:39 smithi001 ceph-mon[29099]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:38:40.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:38:39 smithi028 ceph-mon[35823]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:38:40.309 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:38:41.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:38:41 smithi001 ceph-mon[29099]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:38:42.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:42.141+0000 7f3e3f305640 1 -- 172.21.15.1:0/317533490 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e381022c0 msgr2=0x7f3e38102690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:38:42.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:42.141+0000 7f3e3f305640 1 --2- 172.21.15.1:0/317533490 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e381022c0 0x7f3e38102690 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f3e28009c50 tx=0x7f3e28004d10 comp rx=0 tx=0).stop 2024-06-18T02:38:42.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:42.142+0000 7f3e3f305640 1 -- 172.21.15.1:0/317533490 shutdown_connections 2024-06-18T02:38:42.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:42.142+0000 7f3e3f305640 1 --2- 172.21.15.1:0/317533490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e38102c60 0x7f3e380fc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:38:42.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:42.142+0000 7f3e3f305640 1 --2- 172.21.15.1:0/317533490 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e381022c0 0x7f3e38102690 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:38:42.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:42.142+0000 7f3e3f305640 1 -- 172.21.15.1:0/317533490 >> 172.21.15.1:0/317533490 conn(0x7f3e380f7e50 msgr2=0x7f3e380fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:38:42.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:42.142+0000 7f3e3f305640 1 -- 172.21.15.1:0/317533490 shutdown_connections 2024-06-18T02:38:42.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:42.142+0000 7f3e3f305640 1 -- 172.21.15.1:0/317533490 wait complete. 2024-06-18T02:38:42.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:42.142+0000 7f3e3f305640 1 Processor -- start 2024-06-18T02:38:42.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:42.142+0000 7f3e3f305640 1 -- start start 2024-06-18T02:38:42.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:42.142+0000 7f3e3f305640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e381022c0 0x7f3e38113330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:38:42.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:42.142+0000 7f3e3f305640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e38102c60 0x7f3e38113870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:38:42.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:42.142+0000 7f3e3f305640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e38113e70 con 0x7f3e38102c60 2024-06-18T02:38:42.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:42.142+0000 7f3e3f305640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e3810d2b0 con 0x7f3e381022c0 2024-06-18T02:38:42.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:42.143+0000 7f3e3c8a0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e38102c60 0x7f3e38113870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:38:42.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:42.143+0000 7f3e3c8a0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e38102c60 0x7f3e38113870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56432/0 (socket says 172.21.15.1:56432) 2024-06-18T02:38:42.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:42.143+0000 7f3e3c8a0640 1 -- 172.21.15.1:0/421538535 learned_addr learned my addr 172.21.15.1:0/421538535 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:38:42.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:42.143+0000 7f3e3d0a1640 1 --2- 172.21.15.1:0/421538535 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e381022c0 0x7f3e38113330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:38:42.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:42.143+0000 7f3e3c8a0640 1 -- 172.21.15.1:0/421538535 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e381022c0 msgr2=0x7f3e38113330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:38:42.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:42.143+0000 7f3e3c8a0640 1 --2- 172.21.15.1:0/421538535 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e381022c0 0x7f3e38113330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:38:42.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:42.143+0000 7f3e3c8a0640 1 -- 172.21.15.1:0/421538535 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e280098b0 con 0x7f3e38102c60 2024-06-18T02:38:42.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:42.143+0000 7f3e3c8a0640 1 --2- 172.21.15.1:0/421538535 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e38102c60 0x7f3e38113870 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c002a50 tx=0x7f3e2c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:38:42.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:42.143+0000 7f3e35ffb640 1 -- 172.21.15.1:0/421538535 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e2c007c30 con 0x7f3e38102c60 2024-06-18T02:38:42.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:42.143+0000 7f3e35ffb640 1 -- 172.21.15.1:0/421538535 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3e2c00f040 con 0x7f3e38102c60 2024-06-18T02:38:42.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:42.143+0000 7f3e3f305640 1 -- 172.21.15.1:0/421538535 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e3810d590 con 0x7f3e38102c60 2024-06-18T02:38:42.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:42.143+0000 7f3e35ffb640 1 -- 172.21.15.1:0/421538535 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e2c00db50 con 0x7f3e38102c60 2024-06-18T02:38:42.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:42.143+0000 7f3e3f305640 1 -- 172.21.15.1:0/421538535 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e3810dab0 con 0x7f3e38102c60 2024-06-18T02:38:42.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:42.144+0000 7f3e3f305640 1 -- 172.21.15.1:0/421538535 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e380fda30 con 0x7f3e38102c60 2024-06-18T02:38:42.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:42.145+0000 7f3e35ffb640 1 -- 172.21.15.1:0/421538535 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7f3e2c01c400 con 0x7f3e38102c60 2024-06-18T02:38:42.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:42.145+0000 7f3e35ffb640 1 --2- 172.21.15.1:0/421538535 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f3e20065430 0x7f3e200678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:38:42.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:42.145+0000 7f3e35ffb640 1 -- 172.21.15.1:0/421538535 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f3e2c081d40 con 0x7f3e38102c60 2024-06-18T02:38:42.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:42.145+0000 7f3e3d0a1640 1 --2- 172.21.15.1:0/421538535 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f3e20065430 0x7f3e200678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:38:42.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:42.146+0000 7f3e3d0a1640 1 --2- 172.21.15.1:0/421538535 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f3e20065430 0x7f3e200678e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f3e28009c20 tx=0x7f3e28004d10 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:38:42.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:42.150+0000 7f3e35ffb640 1 -- 172.21.15.1:0/421538535 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f3e2c0502a0 con 0x7f3e38102c60 2024-06-18T02:38:42.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:38:41 smithi028 ceph-mon[35823]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:38:42.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:42.362+0000 7f3e3f305640 1 -- 172.21.15.1:0/421538535 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f3e38109420 con 0x7f3e38102c60 2024-06-18T02:38:42.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:42.362+0000 7f3e35ffb640 1 -- 172.21.15.1:0/421538535 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f3e2c0538c0 con 0x7f3e38102c60 2024-06-18T02:38:42.363 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:38:42.363 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-18T02:38:42.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:42.364+0000 7f3e3f305640 1 -- 172.21.15.1:0/421538535 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f3e20065430 msgr2=0x7f3e200678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:38:42.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:42.364+0000 7f3e3f305640 1 --2- 172.21.15.1:0/421538535 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f3e20065430 0x7f3e200678e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f3e28009c20 tx=0x7f3e28004d10 comp rx=0 tx=0).stop 2024-06-18T02:38:42.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:42.364+0000 7f3e3f305640 1 -- 172.21.15.1:0/421538535 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e38102c60 msgr2=0x7f3e38113870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:38:42.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:42.364+0000 7f3e3f305640 1 --2- 172.21.15.1:0/421538535 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e38102c60 0x7f3e38113870 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c002a50 tx=0x7f3e2c004600 comp rx=0 tx=0).stop 2024-06-18T02:38:42.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:42.365+0000 7f3e3f305640 1 -- 172.21.15.1:0/421538535 shutdown_connections 2024-06-18T02:38:42.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:42.365+0000 7f3e3f305640 1 --2- 172.21.15.1:0/421538535 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f3e20065430 0x7f3e200678e0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:38:42.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:42.365+0000 7f3e3f305640 1 --2- 172.21.15.1:0/421538535 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e38102c60 0x7f3e38113870 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:38:42.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:42.365+0000 7f3e3f305640 1 --2- 172.21.15.1:0/421538535 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e381022c0 0x7f3e38113330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:38:42.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:42.365+0000 7f3e3f305640 1 -- 172.21.15.1:0/421538535 >> 172.21.15.1:0/421538535 conn(0x7f3e380f7e50 msgr2=0x7f3e380f8dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:38:42.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:42.365+0000 7f3e3f305640 1 -- 172.21.15.1:0/421538535 shutdown_connections 2024-06-18T02:38:42.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:42.365+0000 7f3e3f305640 1 -- 172.21.15.1:0/421538535 wait complete. 2024-06-18T02:38:43.100 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:38:42 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/421538535' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:38:43.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:38:42 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/421538535' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:38:44.127 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:38:43 smithi001 ceph-mon[29099]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:38:44.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:38:43 smithi028 ceph-mon[35823]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:38:46.127 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T02:38:46.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:38:45 smithi001 ceph-mon[29099]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:38:46.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:38:45 smithi028 ceph-mon[35823]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:38:46.406 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:38:47.868 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:38:47 smithi001 ceph-mon[29099]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:38:48.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:48.082+0000 7f267b58a640 1 -- 172.21.15.1:0/555417376 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26740fb570 msgr2=0x7f26740fb940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:38:48.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:48.082+0000 7f267b58a640 1 --2- 172.21.15.1:0/555417376 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26740fb570 0x7f26740fb940 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f2670009c50 tx=0x7f2670004d10 comp rx=0 tx=0).stop 2024-06-18T02:38:48.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:48.082+0000 7f267b58a640 1 -- 172.21.15.1:0/555417376 shutdown_connections 2024-06-18T02:38:48.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:48.082+0000 7f267b58a640 1 --2- 172.21.15.1:0/555417376 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26740fbe80 0x7f26740fcd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:38:48.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:48.082+0000 7f267b58a640 1 --2- 172.21.15.1:0/555417376 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26740fb570 0x7f26740fb940 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:38:48.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:48.082+0000 7f267b58a640 1 -- 172.21.15.1:0/555417376 >> 172.21.15.1:0/555417376 conn(0x7f2674072110 msgr2=0x7f2674072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:38:48.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:48.082+0000 7f267b58a640 1 -- 172.21.15.1:0/555417376 shutdown_connections 2024-06-18T02:38:48.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:48.082+0000 7f267b58a640 1 -- 172.21.15.1:0/555417376 wait complete. 2024-06-18T02:38:48.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:48.083+0000 7f267b58a640 1 Processor -- start 2024-06-18T02:38:48.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:48.083+0000 7f267b58a640 1 -- start start 2024-06-18T02:38:48.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:48.083+0000 7f267b58a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26740fb570 0x7f267410dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:38:48.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:48.083+0000 7f267b58a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26740fbe80 0x7f267410e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:38:48.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:48.083+0000 7f267b58a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f267410eb50 con 0x7f26740fbe80 2024-06-18T02:38:48.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:48.083+0000 7f267b58a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f267410ecc0 con 0x7f26740fb570 2024-06-18T02:38:48.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:48.083+0000 7f2678b25640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26740fbe80 0x7f267410e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:38:48.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:48.083+0000 7f2678b25640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26740fbe80 0x7f267410e500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53060/0 (socket says 172.21.15.1:53060) 2024-06-18T02:38:48.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:48.083+0000 7f2678b25640 1 -- 172.21.15.1:0/3639047810 learned_addr learned my addr 172.21.15.1:0/3639047810 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:38:48.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:48.083+0000 7f2679326640 1 --2- 172.21.15.1:0/3639047810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26740fb570 0x7f267410dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:38:48.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:48.083+0000 7f2678b25640 1 -- 172.21.15.1:0/3639047810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26740fb570 msgr2=0x7f267410dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:38:48.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:48.083+0000 7f2678b25640 1 --2- 172.21.15.1:0/3639047810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26740fb570 0x7f267410dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:38:48.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:48.083+0000 7f2678b25640 1 -- 172.21.15.1:0/3639047810 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26700098b0 con 0x7f26740fbe80 2024-06-18T02:38:48.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:48.084+0000 7f2678b25640 1 --2- 172.21.15.1:0/3639047810 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26740fbe80 0x7f267410e500 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f266400c790 tx=0x7f26640043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:38:48.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:48.084+0000 7f2669ffb640 1 -- 172.21.15.1:0/3639047810 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f266400a820 con 0x7f26740fbe80 2024-06-18T02:38:48.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:48.084+0000 7f2669ffb640 1 -- 172.21.15.1:0/3639047810 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f266400ae50 con 0x7f26740fbe80 2024-06-18T02:38:48.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:48.084+0000 7f267b58a640 1 -- 172.21.15.1:0/3639047810 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2674117030 con 0x7f26740fbe80 2024-06-18T02:38:48.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:48.084+0000 7f2669ffb640 1 -- 172.21.15.1:0/3639047810 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f26640115e0 con 0x7f26740fbe80 2024-06-18T02:38:48.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:48.084+0000 7f267b58a640 1 -- 172.21.15.1:0/3639047810 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26741174c0 con 0x7f26740fbe80 2024-06-18T02:38:48.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:48.085+0000 7f267b58a640 1 -- 172.21.15.1:0/3639047810 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26740fc920 con 0x7f26740fbe80 2024-06-18T02:38:48.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:48.085+0000 7f2669ffb640 1 -- 172.21.15.1:0/3639047810 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7f2664012400 con 0x7f26740fbe80 2024-06-18T02:38:48.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:48.086+0000 7f2669ffb640 1 --2- 172.21.15.1:0/3639047810 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f2660065430 0x7f26600678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:38:48.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:48.086+0000 7f2669ffb640 1 -- 172.21.15.1:0/3639047810 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f2664082530 con 0x7f26740fbe80 2024-06-18T02:38:48.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:48.086+0000 7f2679326640 1 --2- 172.21.15.1:0/3639047810 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f2660065430 0x7f26600678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:38:48.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:48.086+0000 7f2679326640 1 --2- 172.21.15.1:0/3639047810 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f2660065430 0x7f26600678e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f2670009c20 tx=0x7f2670004d10 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:38:48.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:48.090+0000 7f2669ffb640 1 -- 172.21.15.1:0/3639047810 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f2664050a80 con 0x7f26740fbe80 2024-06-18T02:38:48.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:38:47 smithi028 ceph-mon[35823]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:38:48.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:48.298+0000 7f267b58a640 1 -- 172.21.15.1:0/3639047810 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f2674072c20 con 0x7f26740fbe80 2024-06-18T02:38:48.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:48.299+0000 7f2669ffb640 1 -- 172.21.15.1:0/3639047810 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f26640540a0 con 0x7f26740fbe80 2024-06-18T02:38:48.299 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:38:48.299 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-18T02:38:48.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:48.300+0000 7f267b58a640 1 -- 172.21.15.1:0/3639047810 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f2660065430 msgr2=0x7f26600678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:38:48.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:48.300+0000 7f267b58a640 1 --2- 172.21.15.1:0/3639047810 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f2660065430 0x7f26600678e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f2670009c20 tx=0x7f2670004d10 comp rx=0 tx=0).stop 2024-06-18T02:38:48.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:48.300+0000 7f267b58a640 1 -- 172.21.15.1:0/3639047810 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26740fbe80 msgr2=0x7f267410e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:38:48.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:48.300+0000 7f267b58a640 1 --2- 172.21.15.1:0/3639047810 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26740fbe80 0x7f267410e500 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f266400c790 tx=0x7f26640043c0 comp rx=0 tx=0).stop 2024-06-18T02:38:48.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:48.300+0000 7f267b58a640 1 -- 172.21.15.1:0/3639047810 shutdown_connections 2024-06-18T02:38:48.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:48.300+0000 7f267b58a640 1 --2- 172.21.15.1:0/3639047810 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f2660065430 0x7f26600678e0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:38:48.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:48.300+0000 7f267b58a640 1 --2- 172.21.15.1:0/3639047810 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26740fbe80 0x7f267410e500 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:38:48.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:48.300+0000 7f267b58a640 1 --2- 172.21.15.1:0/3639047810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26740fb570 0x7f267410dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:38:48.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:48.300+0000 7f267b58a640 1 -- 172.21.15.1:0/3639047810 >> 172.21.15.1:0/3639047810 conn(0x7f2674072110 msgr2=0x7f26740f85d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:38:48.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:48.301+0000 7f267b58a640 1 -- 172.21.15.1:0/3639047810 shutdown_connections 2024-06-18T02:38:48.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:48.301+0000 7f267b58a640 1 -- 172.21.15.1:0/3639047810 wait complete. 2024-06-18T02:38:48.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:38:48 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/3639047810' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:38:49.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:38:48 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/3639047810' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:38:50.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:38:49 smithi001 ceph-mon[29099]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:38:50.139 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:38:49 smithi028 ceph-mon[35823]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:38:51.916 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T02:38:51.941 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:38:51 smithi001 ceph-mon[29099]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:38:51.941 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:38:51 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:38:52.182 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:38:52.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:38:51 smithi028 ceph-mon[35823]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:38:52.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:38:51 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:38:53.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:38:53 smithi001 ceph-mon[29099]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:38:53.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:53.907+0000 7f8daae6f640 1 -- 172.21.15.1:0/806634712 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8da40fdeb0 msgr2=0x7f8da40fe2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:38:53.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:53.907+0000 7f8daae6f640 1 --2- 172.21.15.1:0/806634712 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8da40fdeb0 0x7f8da40fe2e0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c0048d0 tx=0x7f8d8c009da0 comp rx=0 tx=0).stop 2024-06-18T02:38:53.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:53.907+0000 7f8daae6f640 1 -- 172.21.15.1:0/806634712 shutdown_connections 2024-06-18T02:38:53.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:53.907+0000 7f8daae6f640 1 --2- 172.21.15.1:0/806634712 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8da40fdeb0 0x7f8da40fe2e0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:38:53.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:53.907+0000 7f8daae6f640 1 --2- 172.21.15.1:0/806634712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8da40650a0 0x7f8da4065470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:38:53.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:53.907+0000 7f8daae6f640 1 -- 172.21.15.1:0/806634712 >> 172.21.15.1:0/806634712 conn(0x7f8da4072110 msgr2=0x7f8da4072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:38:53.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:53.907+0000 7f8daae6f640 1 -- 172.21.15.1:0/806634712 shutdown_connections 2024-06-18T02:38:53.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:53.907+0000 7f8daae6f640 1 -- 172.21.15.1:0/806634712 wait complete. 2024-06-18T02:38:53.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:53.907+0000 7f8daae6f640 1 Processor -- start 2024-06-18T02:38:53.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:53.907+0000 7f8daae6f640 1 -- start start 2024-06-18T02:38:53.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:53.908+0000 7f8daae6f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8da40650a0 0x7f8da41151d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:38:53.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:53.908+0000 7f8daae6f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8da40fdeb0 0x7f8da4115710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:38:53.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:53.908+0000 7f8daae6f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da4115d10 con 0x7f8da40fdeb0 2024-06-18T02:38:53.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:53.908+0000 7f8daae6f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da410f150 con 0x7f8da40650a0 2024-06-18T02:38:53.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:53.908+0000 7f8da3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8da40fdeb0 0x7f8da4115710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:38:53.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:53.908+0000 7f8da3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8da40fdeb0 0x7f8da4115710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54414/0 (socket says 172.21.15.1:54414) 2024-06-18T02:38:53.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:53.908+0000 7f8da3fff640 1 -- 172.21.15.1:0/3648257994 learned_addr learned my addr 172.21.15.1:0/3648257994 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:38:53.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:53.908+0000 7f8da8c0b640 1 --2- 172.21.15.1:0/3648257994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8da40650a0 0x7f8da41151d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:38:53.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:53.908+0000 7f8da3fff640 1 -- 172.21.15.1:0/3648257994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8da40650a0 msgr2=0x7f8da41151d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:38:53.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:53.908+0000 7f8da3fff640 1 --2- 172.21.15.1:0/3648257994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8da40650a0 0x7f8da41151d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:38:53.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:53.908+0000 7f8da3fff640 1 -- 172.21.15.1:0/3648257994 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d8c005040 con 0x7f8da40fdeb0 2024-06-18T02:38:53.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:53.908+0000 7f8da8c0b640 1 --2- 172.21.15.1:0/3648257994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8da40650a0 0x7f8da41151d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-18T02:38:53.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:53.908+0000 7f8da3fff640 1 --2- 172.21.15.1:0/3648257994 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8da40fdeb0 0x7f8da4115710 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c003080 tx=0x7f8d8c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:38:53.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:53.909+0000 7f8da17fa640 1 -- 172.21.15.1:0/3648257994 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d8c038070 con 0x7f8da40fdeb0 2024-06-18T02:38:53.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:53.909+0000 7f8da17fa640 1 -- 172.21.15.1:0/3648257994 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8d8c008ee0 con 0x7f8da40fdeb0 2024-06-18T02:38:53.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:53.909+0000 7f8daae6f640 1 -- 172.21.15.1:0/3648257994 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8da410f3d0 con 0x7f8da40fdeb0 2024-06-18T02:38:53.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:53.909+0000 7f8da17fa640 1 -- 172.21.15.1:0/3648257994 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d8c03d870 con 0x7f8da40fdeb0 2024-06-18T02:38:53.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:53.909+0000 7f8daae6f640 1 -- 172.21.15.1:0/3648257994 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8da410f910 con 0x7f8da40fdeb0 2024-06-18T02:38:53.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:53.910+0000 7f8daae6f640 1 -- 172.21.15.1:0/3648257994 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8da40fda50 con 0x7f8da40fdeb0 2024-06-18T02:38:53.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:53.910+0000 7f8da17fa640 1 -- 172.21.15.1:0/3648257994 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7f8d8c030020 con 0x7f8da40fdeb0 2024-06-18T02:38:53.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:53.911+0000 7f8da17fa640 1 --2- 172.21.15.1:0/3648257994 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f8d90065430 0x7f8d900678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:38:53.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:53.911+0000 7f8da17fa640 1 -- 172.21.15.1:0/3648257994 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f8d8c0a2140 con 0x7f8da40fdeb0 2024-06-18T02:38:53.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:53.911+0000 7f8da8c0b640 1 --2- 172.21.15.1:0/3648257994 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f8d90065430 0x7f8d900678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:38:53.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:53.911+0000 7f8da8c0b640 1 --2- 172.21.15.1:0/3648257994 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f8d90065430 0x7f8d900678e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f8d9c004dd0 tx=0x7f8d9c009040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:38:53.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:53.915+0000 7f8da17fa640 1 -- 172.21.15.1:0/3648257994 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f8d8c070710 con 0x7f8da40fdeb0 2024-06-18T02:38:54.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:54.123+0000 7f8daae6f640 1 -- 172.21.15.1:0/3648257994 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f8da4110840 con 0x7f8da40fdeb0 2024-06-18T02:38:54.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:54.124+0000 7f8da17fa640 1 -- 172.21.15.1:0/3648257994 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f8d8c073d30 con 0x7f8da40fdeb0 2024-06-18T02:38:54.124 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:38:54.124 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-18T02:38:54.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:54.126+0000 7f8daae6f640 1 -- 172.21.15.1:0/3648257994 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f8d90065430 msgr2=0x7f8d900678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:38:54.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:54.126+0000 7f8daae6f640 1 --2- 172.21.15.1:0/3648257994 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f8d90065430 0x7f8d900678e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f8d9c004dd0 tx=0x7f8d9c009040 comp rx=0 tx=0).stop 2024-06-18T02:38:54.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:54.126+0000 7f8daae6f640 1 -- 172.21.15.1:0/3648257994 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8da40fdeb0 msgr2=0x7f8da4115710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:38:54.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:54.126+0000 7f8daae6f640 1 --2- 172.21.15.1:0/3648257994 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8da40fdeb0 0x7f8da4115710 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c003080 tx=0x7f8d8c009da0 comp rx=0 tx=0).stop 2024-06-18T02:38:54.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:54.126+0000 7f8daae6f640 1 -- 172.21.15.1:0/3648257994 shutdown_connections 2024-06-18T02:38:54.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:54.126+0000 7f8daae6f640 1 --2- 172.21.15.1:0/3648257994 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f8d90065430 0x7f8d900678e0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:38:54.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:54.126+0000 7f8daae6f640 1 --2- 172.21.15.1:0/3648257994 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8da40fdeb0 0x7f8da4115710 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:38:54.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:54.126+0000 7f8daae6f640 1 --2- 172.21.15.1:0/3648257994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8da40650a0 0x7f8da41151d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:38:54.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:54.126+0000 7f8daae6f640 1 -- 172.21.15.1:0/3648257994 >> 172.21.15.1:0/3648257994 conn(0x7f8da4072110 msgr2=0x7f8da40f96c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:38:54.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:54.126+0000 7f8daae6f640 1 -- 172.21.15.1:0/3648257994 shutdown_connections 2024-06-18T02:38:54.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:54.126+0000 7f8daae6f640 1 -- 172.21.15.1:0/3648257994 wait complete. 2024-06-18T02:38:54.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:38:53 smithi028 ceph-mon[35823]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:38:55.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:38:54 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/3648257994' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:38:55.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:38:54 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/3648257994' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:38:56.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:38:55 smithi001 ceph-mon[29099]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:38:56.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:38:55 smithi028 ceph-mon[35823]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:38:57.748 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T02:38:58.024 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:38:58.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:38:57 smithi001 ceph-mon[29099]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:38:58.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:38:57 smithi028 ceph-mon[35823]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:38:59.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:59.730+0000 7f9571da2640 1 -- 172.21.15.1:0/2555559687 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f956c102c40 msgr2=0x7f956c0fc2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:38:59.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:59.730+0000 7f9571da2640 1 --2- 172.21.15.1:0/2555559687 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f956c102c40 0x7f956c0fc2d0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f95540048d0 tx=0x7f9554009da0 comp rx=0 tx=0).stop 2024-06-18T02:38:59.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:59.731+0000 7f9571da2640 1 -- 172.21.15.1:0/2555559687 shutdown_connections 2024-06-18T02:38:59.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:59.731+0000 7f9571da2640 1 --2- 172.21.15.1:0/2555559687 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f956c102c40 0x7f956c0fc2d0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:38:59.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:59.731+0000 7f9571da2640 1 --2- 172.21.15.1:0/2555559687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f956c1022a0 0x7f956c102670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:38:59.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:59.731+0000 7f9571da2640 1 -- 172.21.15.1:0/2555559687 >> 172.21.15.1:0/2555559687 conn(0x7f956c0f7e50 msgr2=0x7f956c0fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:38:59.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:59.731+0000 7f9571da2640 1 -- 172.21.15.1:0/2555559687 shutdown_connections 2024-06-18T02:38:59.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:59.731+0000 7f9571da2640 1 -- 172.21.15.1:0/2555559687 wait complete. 2024-06-18T02:38:59.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:59.731+0000 7f9571da2640 1 Processor -- start 2024-06-18T02:38:59.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:59.731+0000 7f9571da2640 1 -- start start 2024-06-18T02:38:59.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:59.732+0000 7f9571da2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f956c1022a0 0x7f956c1130b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:38:59.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:59.732+0000 7f9571da2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f956c102c40 0x7f956c1135f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:38:59.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:59.732+0000 7f9571da2640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f956c113c80 con 0x7f956c102c40 2024-06-18T02:38:59.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:59.732+0000 7f9571da2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f956c10d030 con 0x7f956c1022a0 2024-06-18T02:38:59.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:59.732+0000 7f956affd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f956c102c40 0x7f956c1135f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:38:59.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:59.732+0000 7f956affd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f956c102c40 0x7f956c1135f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54432/0 (socket says 172.21.15.1:54432) 2024-06-18T02:38:59.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:59.732+0000 7f956affd640 1 -- 172.21.15.1:0/3568799950 learned_addr learned my addr 172.21.15.1:0/3568799950 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:38:59.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:59.732+0000 7f956b7fe640 1 --2- 172.21.15.1:0/3568799950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f956c1022a0 0x7f956c1130b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:38:59.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:59.732+0000 7f956affd640 1 -- 172.21.15.1:0/3568799950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f956c1022a0 msgr2=0x7f956c1130b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:38:59.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:59.732+0000 7f956affd640 1 --2- 172.21.15.1:0/3568799950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f956c1022a0 0x7f956c1130b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:38:59.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:59.732+0000 7f956affd640 1 -- 172.21.15.1:0/3568799950 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9554005040 con 0x7f956c102c40 2024-06-18T02:38:59.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:59.732+0000 7f956affd640 1 --2- 172.21.15.1:0/3568799950 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f956c102c40 0x7f956c1135f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f9554003080 tx=0x7f9554009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:38:59.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:59.732+0000 7f9563fff640 1 -- 172.21.15.1:0/3568799950 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9554038070 con 0x7f956c102c40 2024-06-18T02:38:59.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:59.732+0000 7f9563fff640 1 -- 172.21.15.1:0/3568799950 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9554008ee0 con 0x7f956c102c40 2024-06-18T02:38:59.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:59.732+0000 7f9571da2640 1 -- 172.21.15.1:0/3568799950 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f956c10d2b0 con 0x7f956c102c40 2024-06-18T02:38:59.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:59.733+0000 7f9563fff640 1 -- 172.21.15.1:0/3568799950 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f955403d870 con 0x7f956c102c40 2024-06-18T02:38:59.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:59.733+0000 7f9571da2640 1 -- 172.21.15.1:0/3568799950 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f956c10d770 con 0x7f956c102c40 2024-06-18T02:38:59.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:59.734+0000 7f9571da2640 1 -- 172.21.15.1:0/3568799950 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f956c0fda10 con 0x7f956c102c40 2024-06-18T02:38:59.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:59.734+0000 7f9563fff640 1 -- 172.21.15.1:0/3568799950 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7f9554030020 con 0x7f956c102c40 2024-06-18T02:38:59.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:59.734+0000 7f9563fff640 1 --2- 172.21.15.1:0/3568799950 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f9558065430 0x7f95580678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:38:59.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:59.734+0000 7f9563fff640 1 -- 172.21.15.1:0/3568799950 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f95540a2180 con 0x7f956c102c40 2024-06-18T02:38:59.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:59.734+0000 7f956b7fe640 1 --2- 172.21.15.1:0/3568799950 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f9558065430 0x7f95580678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:38:59.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:59.735+0000 7f956b7fe640 1 --2- 172.21.15.1:0/3568799950 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f9558065430 0x7f95580678e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f955c007540 tx=0x7f955c009780 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:38:59.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:59.739+0000 7f9563fff640 1 -- 172.21.15.1:0/3568799950 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f95540706b0 con 0x7f956c102c40 2024-06-18T02:38:59.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:59.948+0000 7f9571da2640 1 -- 172.21.15.1:0/3568799950 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f956c109490 con 0x7f956c102c40 2024-06-18T02:38:59.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:59.948+0000 7f9563fff640 1 -- 172.21.15.1:0/3568799950 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f9554073cd0 con 0x7f956c102c40 2024-06-18T02:38:59.948 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:38:59.949 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-18T02:38:59.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:59.950+0000 7f9571da2640 1 -- 172.21.15.1:0/3568799950 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f9558065430 msgr2=0x7f95580678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:38:59.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:59.950+0000 7f9571da2640 1 --2- 172.21.15.1:0/3568799950 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f9558065430 0x7f95580678e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f955c007540 tx=0x7f955c009780 comp rx=0 tx=0).stop 2024-06-18T02:38:59.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:59.950+0000 7f9571da2640 1 -- 172.21.15.1:0/3568799950 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f956c102c40 msgr2=0x7f956c1135f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:38:59.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:59.950+0000 7f9571da2640 1 --2- 172.21.15.1:0/3568799950 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f956c102c40 0x7f956c1135f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f9554003080 tx=0x7f9554009da0 comp rx=0 tx=0).stop 2024-06-18T02:38:59.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:59.951+0000 7f9571da2640 1 -- 172.21.15.1:0/3568799950 shutdown_connections 2024-06-18T02:38:59.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:59.951+0000 7f9571da2640 1 --2- 172.21.15.1:0/3568799950 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f9558065430 0x7f95580678e0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:38:59.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:59.951+0000 7f9571da2640 1 --2- 172.21.15.1:0/3568799950 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f956c102c40 0x7f956c1135f0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:38:59.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:59.951+0000 7f9571da2640 1 --2- 172.21.15.1:0/3568799950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f956c1022a0 0x7f956c1130b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:38:59.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:59.951+0000 7f9571da2640 1 -- 172.21.15.1:0/3568799950 >> 172.21.15.1:0/3568799950 conn(0x7f956c0f7e50 msgr2=0x7f956c0f8d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:38:59.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:59.951+0000 7f9571da2640 1 -- 172.21.15.1:0/3568799950 shutdown_connections 2024-06-18T02:38:59.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:38:59.951+0000 7f9571da2640 1 -- 172.21.15.1:0/3568799950 wait complete. 2024-06-18T02:39:00.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:38:59 smithi001 ceph-mon[29099]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:39:00.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:38:59 smithi028 ceph-mon[35823]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:39:01.123 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:39:00 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/3568799950' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:39:01.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:39:00 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/3568799950' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:39:02.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:39:01 smithi001 ceph-mon[29099]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:39:02.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:39:01 smithi028 ceph-mon[35823]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:39:03.755 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T02:39:04.033 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:39:04.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:39:03 smithi001 ceph-mon[29099]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:39:04.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:39:03 smithi028 ceph-mon[35823]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:39:05.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:05.692+0000 7ffbdd98c640 1 -- 172.21.15.1:0/785713014 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffbd8101ba0 msgr2=0x7ffbd80fb230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:39:05.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:05.692+0000 7ffbdd98c640 1 --2- 172.21.15.1:0/785713014 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffbd8101ba0 0x7ffbd80fb230 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7ffbc00048d0 tx=0x7ffbc0009da0 comp rx=0 tx=0).stop 2024-06-18T02:39:05.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:05.692+0000 7ffbdd98c640 1 -- 172.21.15.1:0/785713014 shutdown_connections 2024-06-18T02:39:05.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:05.692+0000 7ffbdd98c640 1 --2- 172.21.15.1:0/785713014 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffbd8101ba0 0x7ffbd80fb230 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:39:05.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:05.692+0000 7ffbdd98c640 1 --2- 172.21.15.1:0/785713014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbd8101200 0x7ffbd81015d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:39:05.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:05.692+0000 7ffbdd98c640 1 -- 172.21.15.1:0/785713014 >> 172.21.15.1:0/785713014 conn(0x7ffbd80f6df0 msgr2=0x7ffbd80f9200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:39:05.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:05.693+0000 7ffbdd98c640 1 -- 172.21.15.1:0/785713014 shutdown_connections 2024-06-18T02:39:05.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:05.693+0000 7ffbdd98c640 1 -- 172.21.15.1:0/785713014 wait complete. 2024-06-18T02:39:05.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:05.693+0000 7ffbdd98c640 1 Processor -- start 2024-06-18T02:39:05.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:05.693+0000 7ffbdd98c640 1 -- start start 2024-06-18T02:39:05.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:05.693+0000 7ffbdd98c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffbd8101200 0x7ffbd810bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:39:05.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:05.693+0000 7ffbdd98c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbd8101ba0 0x7ffbd810c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:39:05.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:05.693+0000 7ffbdd98c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd810c860 con 0x7ffbd8101200 2024-06-18T02:39:05.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:05.693+0000 7ffbdd98c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd810c9a0 con 0x7ffbd8101ba0 2024-06-18T02:39:05.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:05.694+0000 7ffbd6ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffbd8101200 0x7ffbd810bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:39:05.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:05.694+0000 7ffbd6ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffbd8101200 0x7ffbd810bde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55394/0 (socket says 172.21.15.1:55394) 2024-06-18T02:39:05.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:05.694+0000 7ffbd6ffd640 1 -- 172.21.15.1:0/1160172534 learned_addr learned my addr 172.21.15.1:0/1160172534 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:39:05.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:05.694+0000 7ffbd67fc640 1 --2- 172.21.15.1:0/1160172534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbd8101ba0 0x7ffbd810c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:39:05.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:05.694+0000 7ffbd6ffd640 1 -- 172.21.15.1:0/1160172534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbd8101ba0 msgr2=0x7ffbd810c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:39:05.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:05.694+0000 7ffbd6ffd640 1 --2- 172.21.15.1:0/1160172534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbd8101ba0 0x7ffbd810c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:39:05.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:05.694+0000 7ffbd6ffd640 1 -- 172.21.15.1:0/1160172534 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbc0005040 con 0x7ffbd8101200 2024-06-18T02:39:05.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:05.694+0000 7ffbd6ffd640 1 --2- 172.21.15.1:0/1160172534 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffbd8101200 0x7ffbd810bde0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7ffbc8002a50 tx=0x7ffbc8004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:39:05.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:05.694+0000 7ffbcf7fe640 1 -- 172.21.15.1:0/1160172534 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffbc8007c30 con 0x7ffbd8101200 2024-06-18T02:39:05.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:05.694+0000 7ffbcf7fe640 1 -- 172.21.15.1:0/1160172534 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffbc800f040 con 0x7ffbd8101200 2024-06-18T02:39:05.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:05.694+0000 7ffbdd98c640 1 -- 172.21.15.1:0/1160172534 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbd8119450 con 0x7ffbd8101200 2024-06-18T02:39:05.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:05.694+0000 7ffbcf7fe640 1 -- 172.21.15.1:0/1160172534 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffbc800dad0 con 0x7ffbd8101200 2024-06-18T02:39:05.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:05.694+0000 7ffbdd98c640 1 -- 172.21.15.1:0/1160172534 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbd8119800 con 0x7ffbd8101200 2024-06-18T02:39:05.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:05.695+0000 7ffbdd98c640 1 -- 172.21.15.1:0/1160172534 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbd80fc970 con 0x7ffbd8101200 2024-06-18T02:39:05.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:05.696+0000 7ffbcf7fe640 1 -- 172.21.15.1:0/1160172534 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7ffbc800dc70 con 0x7ffbd8101200 2024-06-18T02:39:05.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:05.696+0000 7ffbcf7fe640 1 --2- 172.21.15.1:0/1160172534 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7ffbc4065430 0x7ffbc40678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:39:05.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:05.697+0000 7ffbcf7fe640 1 -- 172.21.15.1:0/1160172534 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7ffbc8081b50 con 0x7ffbd8101200 2024-06-18T02:39:05.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:05.697+0000 7ffbd67fc640 1 --2- 172.21.15.1:0/1160172534 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7ffbc4065430 0x7ffbc40678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:39:05.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:05.697+0000 7ffbd67fc640 1 --2- 172.21.15.1:0/1160172534 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7ffbc4065430 0x7ffbc40678e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ffbc0009f80 tx=0x7ffbc0035040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:39:05.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:05.701+0000 7ffbcf7fe640 1 -- 172.21.15.1:0/1160172534 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7ffbc804cdf0 con 0x7ffbd8101200 2024-06-18T02:39:05.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:05.911+0000 7ffbdd98c640 1 -- 172.21.15.1:0/1160172534 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ffbd810cb60 con 0x7ffbd8101200 2024-06-18T02:39:05.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:05.911+0000 7ffbcf7fe640 1 -- 172.21.15.1:0/1160172534 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7ffbc80501f0 con 0x7ffbd8101200 2024-06-18T02:39:05.911 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:39:05.911 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-18T02:39:05.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:05.913+0000 7ffbdd98c640 1 -- 172.21.15.1:0/1160172534 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7ffbc4065430 msgr2=0x7ffbc40678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:39:05.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:05.913+0000 7ffbdd98c640 1 --2- 172.21.15.1:0/1160172534 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7ffbc4065430 0x7ffbc40678e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ffbc0009f80 tx=0x7ffbc0035040 comp rx=0 tx=0).stop 2024-06-18T02:39:05.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:05.913+0000 7ffbdd98c640 1 -- 172.21.15.1:0/1160172534 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffbd8101200 msgr2=0x7ffbd810bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:39:05.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:05.913+0000 7ffbdd98c640 1 --2- 172.21.15.1:0/1160172534 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffbd8101200 0x7ffbd810bde0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7ffbc8002a50 tx=0x7ffbc8004600 comp rx=0 tx=0).stop 2024-06-18T02:39:05.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:05.913+0000 7ffbdd98c640 1 -- 172.21.15.1:0/1160172534 shutdown_connections 2024-06-18T02:39:05.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:05.913+0000 7ffbdd98c640 1 --2- 172.21.15.1:0/1160172534 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7ffbc4065430 0x7ffbc40678e0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:39:05.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:05.913+0000 7ffbdd98c640 1 --2- 172.21.15.1:0/1160172534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbd8101ba0 0x7ffbd810c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:39:05.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:05.913+0000 7ffbdd98c640 1 --2- 172.21.15.1:0/1160172534 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffbd8101200 0x7ffbd810bde0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:39:05.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:05.913+0000 7ffbdd98c640 1 -- 172.21.15.1:0/1160172534 >> 172.21.15.1:0/1160172534 conn(0x7ffbd80f6df0 msgr2=0x7ffbd80f7d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:39:05.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:05.913+0000 7ffbdd98c640 1 -- 172.21.15.1:0/1160172534 shutdown_connections 2024-06-18T02:39:05.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:05.913+0000 7ffbdd98c640 1 -- 172.21.15.1:0/1160172534 wait complete. 2024-06-18T02:39:06.222 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:39:05 smithi001 ceph-mon[29099]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:39:06.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:39:05 smithi028 ceph-mon[35823]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:39:07.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:39:06 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/1160172534' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:39:07.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:39:06 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:39:07.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:39:06 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/1160172534' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:39:07.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:39:06 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:39:08.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:39:07 smithi028 ceph-mon[35823]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:39:08.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:39:07 smithi001 ceph-mon[29099]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:39:09.627 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T02:39:09.905 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:39:10.027 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:39:09 smithi001 ceph-mon[29099]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:39:10.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:39:09 smithi028 ceph-mon[35823]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:39:11.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:11.555+0000 7f6bd4b62640 1 -- 172.21.15.1:0/4190821800 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bd0065030 msgr2=0x7f6bd0065400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:39:11.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:11.555+0000 7f6bd4b62640 1 --2- 172.21.15.1:0/4190821800 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bd0065030 0x7f6bd0065400 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f6bc00048d0 tx=0x7f6bc0009da0 comp rx=0 tx=0).stop 2024-06-18T02:39:11.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:11.555+0000 7f6bd4b62640 1 -- 172.21.15.1:0/4190821800 shutdown_connections 2024-06-18T02:39:11.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:11.555+0000 7f6bd4b62640 1 --2- 172.21.15.1:0/4190821800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6bd00fdea0 0x7f6bd00fe310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:39:11.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:11.555+0000 7f6bd4b62640 1 --2- 172.21.15.1:0/4190821800 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bd0065030 0x7f6bd0065400 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:39:11.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:11.555+0000 7f6bd4b62640 1 -- 172.21.15.1:0/4190821800 >> 172.21.15.1:0/4190821800 conn(0x7f6bd0072110 msgr2=0x7f6bd0072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:39:11.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:11.555+0000 7f6bd4b62640 1 -- 172.21.15.1:0/4190821800 shutdown_connections 2024-06-18T02:39:11.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:11.555+0000 7f6bd4b62640 1 -- 172.21.15.1:0/4190821800 wait complete. 2024-06-18T02:39:11.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:11.555+0000 7f6bd4b62640 1 Processor -- start 2024-06-18T02:39:11.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:11.556+0000 7f6bd4b62640 1 -- start start 2024-06-18T02:39:11.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:11.556+0000 7f6bd4b62640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bd0065030 0x7f6bd0113040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:39:11.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:11.556+0000 7f6bd4b62640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6bd00fdea0 0x7f6bd0113580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:39:11.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:11.556+0000 7f6bd4b62640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd0113b80 con 0x7f6bd0065030 2024-06-18T02:39:11.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:11.556+0000 7f6bd4b62640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd010d010 con 0x7f6bd00fdea0 2024-06-18T02:39:11.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:11.556+0000 7f6bce59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bd0065030 0x7f6bd0113040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:39:11.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:11.556+0000 7f6bce59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bd0065030 0x7f6bd0113040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55408/0 (socket says 172.21.15.1:55408) 2024-06-18T02:39:11.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:11.556+0000 7f6bce59c640 1 -- 172.21.15.1:0/2494800141 learned_addr learned my addr 172.21.15.1:0/2494800141 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:39:11.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:11.556+0000 7f6bcdd9b640 1 --2- 172.21.15.1:0/2494800141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6bd00fdea0 0x7f6bd0113580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:39:11.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:11.556+0000 7f6bce59c640 1 -- 172.21.15.1:0/2494800141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6bd00fdea0 msgr2=0x7f6bd0113580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:39:11.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:11.556+0000 7f6bce59c640 1 --2- 172.21.15.1:0/2494800141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6bd00fdea0 0x7f6bd0113580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:39:11.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:11.556+0000 7f6bce59c640 1 -- 172.21.15.1:0/2494800141 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bc0005040 con 0x7f6bd0065030 2024-06-18T02:39:11.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:11.556+0000 7f6bcdd9b640 1 --2- 172.21.15.1:0/2494800141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6bd00fdea0 0x7f6bd0113580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-18T02:39:11.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:11.556+0000 7f6bce59c640 1 --2- 172.21.15.1:0/2494800141 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bd0065030 0x7f6bd0113040 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f6bc00043d0 tx=0x7f6bc0009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:39:11.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:11.557+0000 7f6bc6ffd640 1 -- 172.21.15.1:0/2494800141 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6bc0038070 con 0x7f6bd0065030 2024-06-18T02:39:11.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:11.557+0000 7f6bc6ffd640 1 -- 172.21.15.1:0/2494800141 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6bc0032970 con 0x7f6bd0065030 2024-06-18T02:39:11.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:11.557+0000 7f6bd4b62640 1 -- 172.21.15.1:0/2494800141 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bd010d290 con 0x7f6bd0065030 2024-06-18T02:39:11.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:11.557+0000 7f6bc6ffd640 1 -- 172.21.15.1:0/2494800141 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6bc003da60 con 0x7f6bd0065030 2024-06-18T02:39:11.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:11.557+0000 7f6bd4b62640 1 -- 172.21.15.1:0/2494800141 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bd010d750 con 0x7f6bd0065030 2024-06-18T02:39:11.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:11.558+0000 7f6bd4b62640 1 -- 172.21.15.1:0/2494800141 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bd00fda40 con 0x7f6bd0065030 2024-06-18T02:39:11.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:11.558+0000 7f6bc6ffd640 1 -- 172.21.15.1:0/2494800141 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7f6bc0030020 con 0x7f6bd0065030 2024-06-18T02:39:11.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:11.559+0000 7f6bc6ffd640 1 --2- 172.21.15.1:0/2494800141 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f6bbc065430 0x7f6bbc0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:39:11.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:11.559+0000 7f6bc6ffd640 1 -- 172.21.15.1:0/2494800141 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f6bc00a2590 con 0x7f6bd0065030 2024-06-18T02:39:11.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:11.559+0000 7f6bcdd9b640 1 --2- 172.21.15.1:0/2494800141 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f6bbc065430 0x7f6bbc0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:39:11.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:11.559+0000 7f6bcdd9b640 1 --2- 172.21.15.1:0/2494800141 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f6bbc065430 0x7f6bbc0678e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6bb8007fd0 tx=0x7f6bb8009e90 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:39:11.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:11.563+0000 7f6bc6ffd640 1 -- 172.21.15.1:0/2494800141 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f6bc0070a40 con 0x7f6bd0065030 2024-06-18T02:39:11.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:11.770+0000 7f6bd4b62640 1 -- 172.21.15.1:0/2494800141 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f6bd004b480 con 0x7f6bd0065030 2024-06-18T02:39:11.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:11.770+0000 7f6bc6ffd640 1 -- 172.21.15.1:0/2494800141 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f6bc0074060 con 0x7f6bd0065030 2024-06-18T02:39:11.770 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:39:11.771 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-18T02:39:11.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:11.772+0000 7f6bd4b62640 1 -- 172.21.15.1:0/2494800141 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f6bbc065430 msgr2=0x7f6bbc0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:39:11.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:11.772+0000 7f6bd4b62640 1 --2- 172.21.15.1:0/2494800141 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f6bbc065430 0x7f6bbc0678e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6bb8007fd0 tx=0x7f6bb8009e90 comp rx=0 tx=0).stop 2024-06-18T02:39:11.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:11.772+0000 7f6bd4b62640 1 -- 172.21.15.1:0/2494800141 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bd0065030 msgr2=0x7f6bd0113040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:39:11.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:11.772+0000 7f6bd4b62640 1 --2- 172.21.15.1:0/2494800141 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bd0065030 0x7f6bd0113040 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f6bc00043d0 tx=0x7f6bc0009da0 comp rx=0 tx=0).stop 2024-06-18T02:39:11.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:11.772+0000 7f6bd4b62640 1 -- 172.21.15.1:0/2494800141 shutdown_connections 2024-06-18T02:39:11.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:11.772+0000 7f6bd4b62640 1 --2- 172.21.15.1:0/2494800141 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f6bbc065430 0x7f6bbc0678e0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:39:11.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:11.772+0000 7f6bd4b62640 1 --2- 172.21.15.1:0/2494800141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6bd00fdea0 0x7f6bd0113580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:39:11.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:11.772+0000 7f6bd4b62640 1 --2- 172.21.15.1:0/2494800141 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bd0065030 0x7f6bd0113040 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:39:11.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:11.772+0000 7f6bd4b62640 1 -- 172.21.15.1:0/2494800141 >> 172.21.15.1:0/2494800141 conn(0x7f6bd0072110 msgr2=0x7f6bd0105120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:39:11.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:11.772+0000 7f6bd4b62640 1 -- 172.21.15.1:0/2494800141 shutdown_connections 2024-06-18T02:39:11.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:11.772+0000 7f6bd4b62640 1 -- 172.21.15.1:0/2494800141 wait complete. 2024-06-18T02:39:12.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:39:11 smithi028 ceph-mon[35823]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:39:12.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:39:11 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/2494800141' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:39:12.377 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:39:11 smithi001 ceph-mon[29099]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:39:12.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:39:11 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/2494800141' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:39:14.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:39:13 smithi028 ceph-mon[35823]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:39:14.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:39:13 smithi001 ceph-mon[29099]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:39:15.503 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T02:39:15.782 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:39:16.222 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:39:15 smithi001 ceph-mon[29099]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:39:16.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:39:15 smithi028 ceph-mon[35823]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:39:17.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:17.315+0000 7fd1180df640 1 -- 172.21.15.1:0/2688311662 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1101022a0 msgr2=0x7fd110102670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:39:17.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:17.315+0000 7fd1180df640 1 --2- 172.21.15.1:0/2688311662 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1101022a0 0x7fd110102670 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fd100009c50 tx=0x7fd100004d10 comp rx=0 tx=0).stop 2024-06-18T02:39:17.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:17.315+0000 7fd1180df640 1 -- 172.21.15.1:0/2688311662 shutdown_connections 2024-06-18T02:39:17.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:17.315+0000 7fd1180df640 1 --2- 172.21.15.1:0/2688311662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd110102c40 0x7fd1100fc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:39:17.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:17.315+0000 7fd1180df640 1 --2- 172.21.15.1:0/2688311662 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1101022a0 0x7fd110102670 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:39:17.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:17.315+0000 7fd1180df640 1 -- 172.21.15.1:0/2688311662 >> 172.21.15.1:0/2688311662 conn(0x7fd1100f7e50 msgr2=0x7fd1100fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:39:17.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:17.315+0000 7fd1180df640 1 -- 172.21.15.1:0/2688311662 shutdown_connections 2024-06-18T02:39:17.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:17.315+0000 7fd1180df640 1 -- 172.21.15.1:0/2688311662 wait complete. 2024-06-18T02:39:17.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:17.316+0000 7fd1180df640 1 Processor -- start 2024-06-18T02:39:17.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:17.316+0000 7fd1180df640 1 -- start start 2024-06-18T02:39:17.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:17.316+0000 7fd1180df640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1101022a0 0x7fd11006f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:39:17.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:17.316+0000 7fd1180df640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd110102c40 0x7fd11006fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:39:17.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:17.316+0000 7fd1180df640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1100701f0 con 0x7fd110102c40 2024-06-18T02:39:17.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:17.316+0000 7fd1180df640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd110070360 con 0x7fd1101022a0 2024-06-18T02:39:17.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:17.316+0000 7fd11567a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd110102c40 0x7fd11006fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:39:17.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:17.316+0000 7fd11567a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd110102c40 0x7fd11006fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51736/0 (socket says 172.21.15.1:51736) 2024-06-18T02:39:17.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:17.316+0000 7fd11567a640 1 -- 172.21.15.1:0/2608154799 learned_addr learned my addr 172.21.15.1:0/2608154799 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:39:17.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:17.316+0000 7fd115e7b640 1 --2- 172.21.15.1:0/2608154799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1101022a0 0x7fd11006f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:39:17.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:17.316+0000 7fd11567a640 1 -- 172.21.15.1:0/2608154799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1101022a0 msgr2=0x7fd11006f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:39:17.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:17.316+0000 7fd11567a640 1 --2- 172.21.15.1:0/2608154799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1101022a0 0x7fd11006f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:39:17.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:17.316+0000 7fd11567a640 1 -- 172.21.15.1:0/2608154799 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1000098b0 con 0x7fd110102c40 2024-06-18T02:39:17.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:17.317+0000 7fd11567a640 1 --2- 172.21.15.1:0/2608154799 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd110102c40 0x7fd11006fcb0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fd10400c830 tx=0x7fd104004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:39:17.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:17.317+0000 7fd10e7fc640 1 -- 172.21.15.1:0/2608154799 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd10400a850 con 0x7fd110102c40 2024-06-18T02:39:17.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:17.317+0000 7fd10e7fc640 1 -- 172.21.15.1:0/2608154799 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd10400ae80 con 0x7fd110102c40 2024-06-18T02:39:17.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:17.317+0000 7fd1180df640 1 -- 172.21.15.1:0/2608154799 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1100705f0 con 0x7fd110102c40 2024-06-18T02:39:17.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:17.317+0000 7fd10e7fc640 1 -- 172.21.15.1:0/2608154799 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd104012ea0 con 0x7fd110102c40 2024-06-18T02:39:17.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:17.317+0000 7fd1180df640 1 -- 172.21.15.1:0/2608154799 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd110119660 con 0x7fd110102c40 2024-06-18T02:39:17.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:17.318+0000 7fd1180df640 1 -- 172.21.15.1:0/2608154799 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1100fda10 con 0x7fd110102c40 2024-06-18T02:39:17.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:17.318+0000 7fd10e7fc640 1 -- 172.21.15.1:0/2608154799 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7fd104011620 con 0x7fd110102c40 2024-06-18T02:39:17.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:17.319+0000 7fd10e7fc640 1 --2- 172.21.15.1:0/2608154799 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fd0f8065430 0x7fd0f80678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:39:17.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:17.319+0000 7fd10e7fc640 1 -- 172.21.15.1:0/2608154799 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fd104082660 con 0x7fd110102c40 2024-06-18T02:39:17.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:17.319+0000 7fd115e7b640 1 --2- 172.21.15.1:0/2608154799 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fd0f8065430 0x7fd0f80678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:39:17.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:17.319+0000 7fd115e7b640 1 --2- 172.21.15.1:0/2608154799 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fd0f8065430 0x7fd0f80678e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd100009c20 tx=0x7fd100004d10 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:39:17.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:17.323+0000 7fd10e7fc640 1 -- 172.21.15.1:0/2608154799 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fd104050b90 con 0x7fd110102c40 2024-06-18T02:39:17.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:17.531+0000 7fd1180df640 1 -- 172.21.15.1:0/2608154799 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd110070e60 con 0x7fd110102c40 2024-06-18T02:39:17.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:17.532+0000 7fd10e7fc640 1 -- 172.21.15.1:0/2608154799 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fd1040541b0 con 0x7fd110102c40 2024-06-18T02:39:17.532 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:39:17.532 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-18T02:39:17.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:17.533+0000 7fd1180df640 1 -- 172.21.15.1:0/2608154799 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fd0f8065430 msgr2=0x7fd0f80678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:39:17.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:17.533+0000 7fd1180df640 1 --2- 172.21.15.1:0/2608154799 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fd0f8065430 0x7fd0f80678e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd100009c20 tx=0x7fd100004d10 comp rx=0 tx=0).stop 2024-06-18T02:39:17.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:17.533+0000 7fd1180df640 1 -- 172.21.15.1:0/2608154799 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd110102c40 msgr2=0x7fd11006fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:39:17.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:17.533+0000 7fd1180df640 1 --2- 172.21.15.1:0/2608154799 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd110102c40 0x7fd11006fcb0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fd10400c830 tx=0x7fd104004430 comp rx=0 tx=0).stop 2024-06-18T02:39:17.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:17.533+0000 7fd1180df640 1 -- 172.21.15.1:0/2608154799 shutdown_connections 2024-06-18T02:39:17.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:17.533+0000 7fd1180df640 1 --2- 172.21.15.1:0/2608154799 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fd0f8065430 0x7fd0f80678e0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:39:17.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:17.533+0000 7fd1180df640 1 --2- 172.21.15.1:0/2608154799 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd110102c40 0x7fd11006fcb0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:39:17.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:17.533+0000 7fd1180df640 1 --2- 172.21.15.1:0/2608154799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1101022a0 0x7fd11006f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:39:17.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:17.533+0000 7fd1180df640 1 -- 172.21.15.1:0/2608154799 >> 172.21.15.1:0/2608154799 conn(0x7fd1100f7e50 msgr2=0x7fd1100f8db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:39:17.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:17.534+0000 7fd1180df640 1 -- 172.21.15.1:0/2608154799 shutdown_connections 2024-06-18T02:39:17.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:17.534+0000 7fd1180df640 1 -- 172.21.15.1:0/2608154799 wait complete. 2024-06-18T02:39:18.169 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:39:17 smithi001 ceph-mon[29099]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:39:18.169 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:39:17 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/2608154799' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:39:18.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:39:17 smithi028 ceph-mon[35823]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:39:18.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:39:17 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/2608154799' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:39:20.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:39:19 smithi028 ceph-mon[35823]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:39:20.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:39:19 smithi001 ceph-mon[29099]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:39:21.171 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T02:39:21.448 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:39:22.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:39:21 smithi001 ceph-mon[29099]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:39:22.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:39:21 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:39:22.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:39:21 smithi028 ceph-mon[35823]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:39:22.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:39:21 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:39:23.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:23.025+0000 7faa2ca38640 1 -- 172.21.15.1:0/1490991706 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa28101310 msgr2=0x7faa281016e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:39:23.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:23.025+0000 7faa2ca38640 1 --2- 172.21.15.1:0/1490991706 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa28101310 0x7faa281016e0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7faa180048d0 tx=0x7faa18009da0 comp rx=0 tx=0).stop 2024-06-18T02:39:23.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:23.025+0000 7faa2ca38640 1 -- 172.21.15.1:0/1490991706 shutdown_connections 2024-06-18T02:39:23.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:23.025+0000 7faa2ca38640 1 --2- 172.21.15.1:0/1490991706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa28101cb0 0x7faa280fb230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:39:23.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:23.025+0000 7faa2ca38640 1 --2- 172.21.15.1:0/1490991706 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa28101310 0x7faa281016e0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:39:23.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:23.025+0000 7faa2ca38640 1 -- 172.21.15.1:0/1490991706 >> 172.21.15.1:0/1490991706 conn(0x7faa280f6d90 msgr2=0x7faa280f91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:39:23.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:23.025+0000 7faa2ca38640 1 -- 172.21.15.1:0/1490991706 shutdown_connections 2024-06-18T02:39:23.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:23.025+0000 7faa2ca38640 1 -- 172.21.15.1:0/1490991706 wait complete. 2024-06-18T02:39:23.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:23.026+0000 7faa2ca38640 1 Processor -- start 2024-06-18T02:39:23.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:23.026+0000 7faa2ca38640 1 -- start start 2024-06-18T02:39:23.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:23.026+0000 7faa2ca38640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa28101310 0x7faa28112020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:39:23.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:23.026+0000 7faa2ca38640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa28101cb0 0x7faa28112560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:39:23.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:23.026+0000 7faa2ca38640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa28112b60 con 0x7faa28101310 2024-06-18T02:39:23.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:23.026+0000 7faa2ca38640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa2810bfa0 con 0x7faa28101cb0 2024-06-18T02:39:23.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:23.026+0000 7faa2659c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa28101310 0x7faa28112020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:39:23.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:23.026+0000 7faa2659c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa28101310 0x7faa28112020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51746/0 (socket says 172.21.15.1:51746) 2024-06-18T02:39:23.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:23.026+0000 7faa2659c640 1 -- 172.21.15.1:0/1388663725 learned_addr learned my addr 172.21.15.1:0/1388663725 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:39:23.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:23.027+0000 7faa25d9b640 1 --2- 172.21.15.1:0/1388663725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa28101cb0 0x7faa28112560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:39:23.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:23.027+0000 7faa2659c640 1 -- 172.21.15.1:0/1388663725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa28101cb0 msgr2=0x7faa28112560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:39:23.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:23.027+0000 7faa2659c640 1 --2- 172.21.15.1:0/1388663725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa28101cb0 0x7faa28112560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:39:23.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:23.027+0000 7faa2659c640 1 -- 172.21.15.1:0/1388663725 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa18005040 con 0x7faa28101310 2024-06-18T02:39:23.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:23.027+0000 7faa2659c640 1 --2- 172.21.15.1:0/1388663725 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa28101310 0x7faa28112020 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7faa18003080 tx=0x7faa18009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:39:23.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:23.027+0000 7faa16ffd640 1 -- 172.21.15.1:0/1388663725 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faa18038070 con 0x7faa28101310 2024-06-18T02:39:23.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:23.027+0000 7faa2ca38640 1 -- 172.21.15.1:0/1388663725 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa2810c220 con 0x7faa28101310 2024-06-18T02:39:23.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:23.027+0000 7faa16ffd640 1 -- 172.21.15.1:0/1388663725 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faa18008ee0 con 0x7faa28101310 2024-06-18T02:39:23.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:23.027+0000 7faa16ffd640 1 -- 172.21.15.1:0/1388663725 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faa1803d870 con 0x7faa28101310 2024-06-18T02:39:23.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:23.027+0000 7faa2ca38640 1 -- 172.21.15.1:0/1388663725 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa2810c760 con 0x7faa28101310 2024-06-18T02:39:23.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:23.028+0000 7faa2ca38640 1 -- 172.21.15.1:0/1388663725 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa280fa9f0 con 0x7faa28101310 2024-06-18T02:39:23.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:23.029+0000 7faa16ffd640 1 -- 172.21.15.1:0/1388663725 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7faa18030020 con 0x7faa28101310 2024-06-18T02:39:23.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:23.029+0000 7faa16ffd640 1 --2- 172.21.15.1:0/1388663725 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7faa10065430 0x7faa100678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:39:23.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:23.029+0000 7faa16ffd640 1 -- 172.21.15.1:0/1388663725 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7faa180a2170 con 0x7faa28101310 2024-06-18T02:39:23.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:23.029+0000 7faa25d9b640 1 --2- 172.21.15.1:0/1388663725 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7faa10065430 0x7faa100678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:39:23.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:23.030+0000 7faa25d9b640 1 --2- 172.21.15.1:0/1388663725 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7faa10065430 0x7faa100678e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7faa2810d8e0 tx=0x7faa1c009040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:39:23.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:23.034+0000 7faa16ffd640 1 -- 172.21.15.1:0/1388663725 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7faa180706a0 con 0x7faa28101310 2024-06-18T02:39:23.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:23.246+0000 7faa2ca38640 1 -- 172.21.15.1:0/1388663725 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7faa2810d690 con 0x7faa28101310 2024-06-18T02:39:23.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:23.246+0000 7faa16ffd640 1 -- 172.21.15.1:0/1388663725 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7faa18073cc0 con 0x7faa28101310 2024-06-18T02:39:23.246 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:39:23.246 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-18T02:39:23.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:23.248+0000 7faa2ca38640 1 -- 172.21.15.1:0/1388663725 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7faa10065430 msgr2=0x7faa100678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:39:23.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:23.248+0000 7faa2ca38640 1 --2- 172.21.15.1:0/1388663725 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7faa10065430 0x7faa100678e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7faa2810d8e0 tx=0x7faa1c009040 comp rx=0 tx=0).stop 2024-06-18T02:39:23.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:23.248+0000 7faa2ca38640 1 -- 172.21.15.1:0/1388663725 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa28101310 msgr2=0x7faa28112020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:39:23.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:23.248+0000 7faa2ca38640 1 --2- 172.21.15.1:0/1388663725 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa28101310 0x7faa28112020 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7faa18003080 tx=0x7faa18009da0 comp rx=0 tx=0).stop 2024-06-18T02:39:23.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:23.248+0000 7faa2ca38640 1 -- 172.21.15.1:0/1388663725 shutdown_connections 2024-06-18T02:39:23.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:23.248+0000 7faa2ca38640 1 --2- 172.21.15.1:0/1388663725 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7faa10065430 0x7faa100678e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:39:23.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:23.248+0000 7faa2ca38640 1 --2- 172.21.15.1:0/1388663725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa28101cb0 0x7faa28112560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:39:23.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:23.248+0000 7faa2ca38640 1 --2- 172.21.15.1:0/1388663725 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa28101310 0x7faa28112020 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:39:23.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:23.248+0000 7faa2ca38640 1 -- 172.21.15.1:0/1388663725 >> 172.21.15.1:0/1388663725 conn(0x7faa280f6d90 msgr2=0x7faa280f7cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:39:23.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:23.248+0000 7faa2ca38640 1 -- 172.21.15.1:0/1388663725 shutdown_connections 2024-06-18T02:39:23.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:23.249+0000 7faa2ca38640 1 -- 172.21.15.1:0/1388663725 wait complete. 2024-06-18T02:39:24.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:39:23 smithi001 ceph-mon[29099]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:39:24.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:39:23 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/1388663725' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:39:24.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:39:23 smithi028 ceph-mon[35823]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:39:24.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:39:23 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/1388663725' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:39:26.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:39:25 smithi028 ceph-mon[35823]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:39:26.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:39:25 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:39:26.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:39:25 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:39:26.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:39:25 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T02:39:26.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:39:25 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:39:26.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:39:25 smithi001 ceph-mon[29099]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:39:26.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:39:25 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:39:26.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:39:25 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:39:26.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:39:25 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T02:39:26.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:39:25 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:39:26.844 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T02:39:27.120 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:39:28.162 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:39:27 smithi001 ceph-mon[29099]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:39:28.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:39:27 smithi028 ceph-mon[35823]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:39:28.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:28.630+0000 7fcfba1c7640 1 -- 172.21.15.1:0/2457078466 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcfb41022f0 msgr2=0x7fcfb41026c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:39:28.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:28.630+0000 7fcfba1c7640 1 --2- 172.21.15.1:0/2457078466 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcfb41022f0 0x7fcfb41026c0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fcfa40048d0 tx=0x7fcfa4009da0 comp rx=0 tx=0).stop 2024-06-18T02:39:28.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:28.630+0000 7fcfba1c7640 1 -- 172.21.15.1:0/2457078466 shutdown_connections 2024-06-18T02:39:28.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:28.630+0000 7fcfba1c7640 1 --2- 172.21.15.1:0/2457078466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfb4102c90 0x7fcfb40fa9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:39:28.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:28.630+0000 7fcfba1c7640 1 --2- 172.21.15.1:0/2457078466 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcfb41022f0 0x7fcfb41026c0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:39:28.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:28.630+0000 7fcfba1c7640 1 -- 172.21.15.1:0/2457078466 >> 172.21.15.1:0/2457078466 conn(0x7fcfb4070a40 msgr2=0x7fcfb4070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:39:28.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:28.630+0000 7fcfba1c7640 1 -- 172.21.15.1:0/2457078466 shutdown_connections 2024-06-18T02:39:28.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:28.630+0000 7fcfba1c7640 1 -- 172.21.15.1:0/2457078466 wait complete. 2024-06-18T02:39:28.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:28.631+0000 7fcfba1c7640 1 Processor -- start 2024-06-18T02:39:28.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:28.631+0000 7fcfba1c7640 1 -- start start 2024-06-18T02:39:28.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:28.631+0000 7fcfba1c7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcfb41022f0 0x7fcfb4115350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:39:28.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:28.631+0000 7fcfba1c7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfb4102c90 0x7fcfb4115890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:39:28.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:28.631+0000 7fcfba1c7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfb4115f20 con 0x7fcfb41022f0 2024-06-18T02:39:28.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:28.631+0000 7fcfba1c7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfb410f2d0 con 0x7fcfb4102c90 2024-06-18T02:39:28.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:28.631+0000 7fcfb37fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcfb41022f0 0x7fcfb4115350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:39:28.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:28.631+0000 7fcfb37fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcfb41022f0 0x7fcfb4115350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59576/0 (socket says 172.21.15.1:59576) 2024-06-18T02:39:28.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:28.631+0000 7fcfb37fe640 1 -- 172.21.15.1:0/3540662491 learned_addr learned my addr 172.21.15.1:0/3540662491 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:39:28.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:28.631+0000 7fcfb2ffd640 1 --2- 172.21.15.1:0/3540662491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfb4102c90 0x7fcfb4115890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:39:28.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:28.632+0000 7fcfb37fe640 1 -- 172.21.15.1:0/3540662491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfb4102c90 msgr2=0x7fcfb4115890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:39:28.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:28.632+0000 7fcfb37fe640 1 --2- 172.21.15.1:0/3540662491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfb4102c90 0x7fcfb4115890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:39:28.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:28.632+0000 7fcfb37fe640 1 -- 172.21.15.1:0/3540662491 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfa4005040 con 0x7fcfb41022f0 2024-06-18T02:39:28.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:28.632+0000 7fcfb37fe640 1 --2- 172.21.15.1:0/3540662491 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcfb41022f0 0x7fcfb4115350 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fcfa4008fc0 tx=0x7fcfa4009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:39:28.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:28.632+0000 7fcf9bfff640 1 -- 172.21.15.1:0/3540662491 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfa4038070 con 0x7fcfb41022f0 2024-06-18T02:39:28.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:28.632+0000 7fcfba1c7640 1 -- 172.21.15.1:0/3540662491 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfb410f550 con 0x7fcfb41022f0 2024-06-18T02:39:28.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:28.632+0000 7fcf9bfff640 1 -- 172.21.15.1:0/3540662491 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfa4032b80 con 0x7fcfb41022f0 2024-06-18T02:39:28.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:28.632+0000 7fcf9bfff640 1 -- 172.21.15.1:0/3540662491 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcfa403dc20 con 0x7fcfb41022f0 2024-06-18T02:39:28.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:28.632+0000 7fcfba1c7640 1 -- 172.21.15.1:0/3540662491 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfb410fa10 con 0x7fcfb41022f0 2024-06-18T02:39:28.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:28.633+0000 7fcfba1c7640 1 -- 172.21.15.1:0/3540662491 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfb4065510 con 0x7fcfb41022f0 2024-06-18T02:39:28.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:28.634+0000 7fcf9bfff640 1 -- 172.21.15.1:0/3540662491 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7fcfa4030020 con 0x7fcfb41022f0 2024-06-18T02:39:28.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:28.634+0000 7fcf9bfff640 1 --2- 172.21.15.1:0/3540662491 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fcf9c065430 0x7fcf9c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:39:28.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:28.634+0000 7fcf9bfff640 1 -- 172.21.15.1:0/3540662491 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fcfa40a2e40 con 0x7fcfb41022f0 2024-06-18T02:39:28.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:28.634+0000 7fcfb2ffd640 1 --2- 172.21.15.1:0/3540662491 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fcf9c065430 0x7fcf9c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:39:28.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:28.635+0000 7fcfb2ffd640 1 --2- 172.21.15.1:0/3540662491 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fcf9c065430 0x7fcf9c0678e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fcfb4110c10 tx=0x7fcfa8009120 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:39:28.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:28.639+0000 7fcf9bfff640 1 -- 172.21.15.1:0/3540662491 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fcfa40713b0 con 0x7fcfb41022f0 2024-06-18T02:39:28.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:28.854+0000 7fcfba1c7640 1 -- 172.21.15.1:0/3540662491 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fcfb404b510 con 0x7fcfb41022f0 2024-06-18T02:39:28.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:28.854+0000 7fcf9bfff640 1 -- 172.21.15.1:0/3540662491 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fcfa40749d0 con 0x7fcfb41022f0 2024-06-18T02:39:28.854 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:39:28.854 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-18T02:39:28.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:28.856+0000 7fcfba1c7640 1 -- 172.21.15.1:0/3540662491 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fcf9c065430 msgr2=0x7fcf9c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:39:28.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:28.856+0000 7fcfba1c7640 1 --2- 172.21.15.1:0/3540662491 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fcf9c065430 0x7fcf9c0678e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fcfb4110c10 tx=0x7fcfa8009120 comp rx=0 tx=0).stop 2024-06-18T02:39:28.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:28.856+0000 7fcfba1c7640 1 -- 172.21.15.1:0/3540662491 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcfb41022f0 msgr2=0x7fcfb4115350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:39:28.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:28.856+0000 7fcfba1c7640 1 --2- 172.21.15.1:0/3540662491 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcfb41022f0 0x7fcfb4115350 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fcfa4008fc0 tx=0x7fcfa4009da0 comp rx=0 tx=0).stop 2024-06-18T02:39:28.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:28.856+0000 7fcfba1c7640 1 -- 172.21.15.1:0/3540662491 shutdown_connections 2024-06-18T02:39:28.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:28.856+0000 7fcfba1c7640 1 --2- 172.21.15.1:0/3540662491 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fcf9c065430 0x7fcf9c0678e0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:39:28.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:28.856+0000 7fcfba1c7640 1 --2- 172.21.15.1:0/3540662491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfb4102c90 0x7fcfb4115890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:39:28.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:28.856+0000 7fcfba1c7640 1 --2- 172.21.15.1:0/3540662491 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcfb41022f0 0x7fcfb4115350 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:39:28.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:28.856+0000 7fcfba1c7640 1 -- 172.21.15.1:0/3540662491 >> 172.21.15.1:0/3540662491 conn(0x7fcfb4070a40 msgr2=0x7fcfb41030a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:39:28.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:28.856+0000 7fcfba1c7640 1 -- 172.21.15.1:0/3540662491 shutdown_connections 2024-06-18T02:39:28.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:28.856+0000 7fcfba1c7640 1 -- 172.21.15.1:0/3540662491 wait complete. 2024-06-18T02:39:29.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:39:28 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/3540662491' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:39:29.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:39:28 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/3540662491' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:39:30.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:39:29 smithi028 ceph-mon[35823]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:39:30.377 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:39:29 smithi001 ceph-mon[29099]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:39:32.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:39:31 smithi028 ceph-mon[35823]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:39:32.377 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:39:31 smithi001 ceph-mon[29099]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:39:32.462 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T02:39:32.740 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:39:34.115 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:39:33 smithi001 ceph-mon[29099]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:39:34.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:39:33 smithi028 ceph-mon[35823]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:39:34.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:34.309+0000 7f3e7b595640 1 -- 172.21.15.1:0/1786939062 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e740fa9e0 msgr2=0x7f3e740fae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:39:34.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:34.309+0000 7f3e7b595640 1 --2- 172.21.15.1:0/1786939062 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e740fa9e0 0x7f3e740fae30 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f3e640048d0 tx=0x7f3e64009da0 comp rx=0 tx=0).stop 2024-06-18T02:39:34.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:34.309+0000 7f3e7b595640 1 -- 172.21.15.1:0/1786939062 shutdown_connections 2024-06-18T02:39:34.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:34.309+0000 7f3e7b595640 1 --2- 172.21.15.1:0/1786939062 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e740fa9e0 0x7f3e740fae30 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:39:34.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:34.309+0000 7f3e7b595640 1 --2- 172.21.15.1:0/1786939062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e740fd100 0x7f3e740fd4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:39:34.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:34.309+0000 7f3e7b595640 1 -- 172.21.15.1:0/1786939062 >> 172.21.15.1:0/1786939062 conn(0x7f3e74070a40 msgr2=0x7f3e74070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:39:34.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:34.309+0000 7f3e7b595640 1 -- 172.21.15.1:0/1786939062 shutdown_connections 2024-06-18T02:39:34.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:34.309+0000 7f3e7b595640 1 -- 172.21.15.1:0/1786939062 wait complete. 2024-06-18T02:39:34.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:34.310+0000 7f3e7b595640 1 Processor -- start 2024-06-18T02:39:34.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:34.310+0000 7f3e7b595640 1 -- start start 2024-06-18T02:39:34.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:34.310+0000 7f3e7b595640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e740fa9e0 0x7f3e741152c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:39:34.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:34.310+0000 7f3e7b595640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e740fd100 0x7f3e74115800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:39:34.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:34.310+0000 7f3e7b595640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e74115e90 con 0x7f3e740fd100 2024-06-18T02:39:34.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:34.310+0000 7f3e7b595640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e7410f240 con 0x7f3e740fa9e0 2024-06-18T02:39:34.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:34.310+0000 7f3e78b30640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e740fd100 0x7f3e74115800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:39:34.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:34.310+0000 7f3e78b30640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e740fd100 0x7f3e74115800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49174/0 (socket says 172.21.15.1:49174) 2024-06-18T02:39:34.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:34.310+0000 7f3e78b30640 1 -- 172.21.15.1:0/620686242 learned_addr learned my addr 172.21.15.1:0/620686242 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:39:34.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:34.311+0000 7f3e79331640 1 --2- 172.21.15.1:0/620686242 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e740fa9e0 0x7f3e741152c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:39:34.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:34.311+0000 7f3e78b30640 1 -- 172.21.15.1:0/620686242 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e740fa9e0 msgr2=0x7f3e741152c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:39:34.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:34.311+0000 7f3e78b30640 1 --2- 172.21.15.1:0/620686242 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e740fa9e0 0x7f3e741152c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:39:34.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:34.311+0000 7f3e78b30640 1 -- 172.21.15.1:0/620686242 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e64005040 con 0x7f3e740fd100 2024-06-18T02:39:34.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:34.311+0000 7f3e79331640 1 --2- 172.21.15.1:0/620686242 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e740fa9e0 0x7f3e741152c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-18T02:39:34.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:34.311+0000 7f3e78b30640 1 --2- 172.21.15.1:0/620686242 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e740fd100 0x7f3e74115800 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f3e64003080 tx=0x7f3e64009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:39:34.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:34.311+0000 7f3e69ffb640 1 -- 172.21.15.1:0/620686242 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e64038070 con 0x7f3e740fd100 2024-06-18T02:39:34.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:34.311+0000 7f3e69ffb640 1 -- 172.21.15.1:0/620686242 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3e64008ee0 con 0x7f3e740fd100 2024-06-18T02:39:34.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:34.311+0000 7f3e69ffb640 1 -- 172.21.15.1:0/620686242 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e6403d870 con 0x7f3e740fd100 2024-06-18T02:39:34.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:34.311+0000 7f3e7b595640 1 -- 172.21.15.1:0/620686242 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e7410f4c0 con 0x7f3e740fd100 2024-06-18T02:39:34.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:34.311+0000 7f3e7b595640 1 -- 172.21.15.1:0/620686242 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e7410f980 con 0x7f3e740fd100 2024-06-18T02:39:34.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:34.312+0000 7f3e7b595640 1 -- 172.21.15.1:0/620686242 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e740fb9f0 con 0x7f3e740fd100 2024-06-18T02:39:34.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:34.313+0000 7f3e69ffb640 1 -- 172.21.15.1:0/620686242 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7f3e64030020 con 0x7f3e740fd100 2024-06-18T02:39:34.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:34.313+0000 7f3e69ffb640 1 --2- 172.21.15.1:0/620686242 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f3e60065430 0x7f3e600678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:39:34.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:34.313+0000 7f3e69ffb640 1 -- 172.21.15.1:0/620686242 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f3e640a21a0 con 0x7f3e740fd100 2024-06-18T02:39:34.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:34.313+0000 7f3e79331640 1 --2- 172.21.15.1:0/620686242 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f3e60065430 0x7f3e600678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:39:34.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:34.314+0000 7f3e79331640 1 --2- 172.21.15.1:0/620686242 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f3e60065430 0x7f3e600678e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3e70007930 tx=0x7f3e70008040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:39:34.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:34.318+0000 7f3e69ffb640 1 -- 172.21.15.1:0/620686242 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f3e640706d0 con 0x7f3e740fd100 2024-06-18T02:39:34.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:34.527+0000 7f3e7b595640 1 -- 172.21.15.1:0/620686242 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f3e7410b700 con 0x7f3e740fd100 2024-06-18T02:39:34.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:34.527+0000 7f3e69ffb640 1 -- 172.21.15.1:0/620686242 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f3e64073cf0 con 0x7f3e740fd100 2024-06-18T02:39:34.527 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:39:34.527 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-18T02:39:34.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:34.529+0000 7f3e7b595640 1 -- 172.21.15.1:0/620686242 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f3e60065430 msgr2=0x7f3e600678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:39:34.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:34.529+0000 7f3e7b595640 1 --2- 172.21.15.1:0/620686242 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f3e60065430 0x7f3e600678e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3e70007930 tx=0x7f3e70008040 comp rx=0 tx=0).stop 2024-06-18T02:39:34.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:34.529+0000 7f3e7b595640 1 -- 172.21.15.1:0/620686242 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e740fd100 msgr2=0x7f3e74115800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:39:34.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:34.529+0000 7f3e7b595640 1 --2- 172.21.15.1:0/620686242 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e740fd100 0x7f3e74115800 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f3e64003080 tx=0x7f3e64009da0 comp rx=0 tx=0).stop 2024-06-18T02:39:34.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:34.529+0000 7f3e7b595640 1 -- 172.21.15.1:0/620686242 shutdown_connections 2024-06-18T02:39:34.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:34.529+0000 7f3e7b595640 1 --2- 172.21.15.1:0/620686242 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f3e60065430 0x7f3e600678e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:39:34.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:34.529+0000 7f3e7b595640 1 --2- 172.21.15.1:0/620686242 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e740fd100 0x7f3e74115800 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:39:34.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:34.529+0000 7f3e7b595640 1 --2- 172.21.15.1:0/620686242 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e740fa9e0 0x7f3e741152c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:39:34.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:34.529+0000 7f3e7b595640 1 -- 172.21.15.1:0/620686242 >> 172.21.15.1:0/620686242 conn(0x7f3e74070a40 msgr2=0x7f3e740f89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:39:34.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:34.529+0000 7f3e7b595640 1 -- 172.21.15.1:0/620686242 shutdown_connections 2024-06-18T02:39:34.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:34.529+0000 7f3e7b595640 1 -- 172.21.15.1:0/620686242 wait complete. 2024-06-18T02:39:35.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:39:34 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/620686242' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:39:35.377 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:39:34 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/620686242' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:39:36.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:39:35 smithi028 ceph-mon[35823]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:39:36.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:39:35 smithi001 ceph-mon[29099]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:39:37.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:39:36 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:39:37.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:39:36 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:39:38.159 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T02:39:38.188 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:39:37 smithi001 ceph-mon[29099]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:39:38.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:39:37 smithi028 ceph-mon[35823]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:39:38.435 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:39:39.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:39.912+0000 7ff42de81640 1 -- 172.21.15.1:0/186538930 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4280fbe80 msgr2=0x7ff4280fcd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:39:39.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:39.912+0000 7ff42de81640 1 --2- 172.21.15.1:0/186538930 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4280fbe80 0x7ff4280fcd80 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7ff41c0048d0 tx=0x7ff41c009da0 comp rx=0 tx=0).stop 2024-06-18T02:39:39.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:39.912+0000 7ff42de81640 1 -- 172.21.15.1:0/186538930 shutdown_connections 2024-06-18T02:39:39.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:39.912+0000 7ff42de81640 1 --2- 172.21.15.1:0/186538930 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4280fbe80 0x7ff4280fcd80 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:39:39.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:39.912+0000 7ff42de81640 1 --2- 172.21.15.1:0/186538930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4280fb570 0x7ff4280fb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:39:39.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:39.912+0000 7ff42de81640 1 -- 172.21.15.1:0/186538930 >> 172.21.15.1:0/186538930 conn(0x7ff428072110 msgr2=0x7ff428072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:39:39.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:39.912+0000 7ff42de81640 1 -- 172.21.15.1:0/186538930 shutdown_connections 2024-06-18T02:39:39.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:39.912+0000 7ff42de81640 1 -- 172.21.15.1:0/186538930 wait complete. 2024-06-18T02:39:39.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:39.912+0000 7ff42de81640 1 Processor -- start 2024-06-18T02:39:39.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:39.913+0000 7ff42de81640 1 -- start start 2024-06-18T02:39:39.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:39.913+0000 7ff42de81640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4280fb570 0x7ff42810e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:39:39.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:39.913+0000 7ff42de81640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4280fbe80 0x7ff42810e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:39:39.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:39.913+0000 7ff42de81640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff42810eb50 con 0x7ff4280fb570 2024-06-18T02:39:39.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:39.913+0000 7ff42de81640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff42810ecc0 con 0x7ff4280fbe80 2024-06-18T02:39:39.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:39.913+0000 7ff4277fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4280fb570 0x7ff42810e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:39:39.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:39.913+0000 7ff4277fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4280fb570 0x7ff42810e0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49188/0 (socket says 172.21.15.1:49188) 2024-06-18T02:39:39.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:39.913+0000 7ff4277fe640 1 -- 172.21.15.1:0/3803053561 learned_addr learned my addr 172.21.15.1:0/3803053561 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:39:39.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:39.913+0000 7ff426ffd640 1 --2- 172.21.15.1:0/3803053561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4280fbe80 0x7ff42810e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:39:39.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:39.913+0000 7ff4277fe640 1 -- 172.21.15.1:0/3803053561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4280fbe80 msgr2=0x7ff42810e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:39:39.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:39.913+0000 7ff4277fe640 1 --2- 172.21.15.1:0/3803053561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4280fbe80 0x7ff42810e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:39:39.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:39.913+0000 7ff4277fe640 1 -- 172.21.15.1:0/3803053561 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff41c005040 con 0x7ff4280fb570 2024-06-18T02:39:39.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:39.913+0000 7ff4277fe640 1 --2- 172.21.15.1:0/3803053561 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4280fb570 0x7ff42810e0d0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7ff41800db00 tx=0x7ff418004e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:39:39.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:39.914+0000 7ff40ffff640 1 -- 172.21.15.1:0/3803053561 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff418017070 con 0x7ff4280fb570 2024-06-18T02:39:39.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:39.914+0000 7ff40ffff640 1 -- 172.21.15.1:0/3803053561 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff41800f040 con 0x7ff4280fb570 2024-06-18T02:39:39.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:39.914+0000 7ff42de81640 1 -- 172.21.15.1:0/3803053561 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff428116fa0 con 0x7ff4280fb570 2024-06-18T02:39:39.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:39.914+0000 7ff40ffff640 1 -- 172.21.15.1:0/3803053561 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff418003a70 con 0x7ff4280fb570 2024-06-18T02:39:39.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:39.914+0000 7ff42de81640 1 -- 172.21.15.1:0/3803053561 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4281174b0 con 0x7ff4280fb570 2024-06-18T02:39:39.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:39.915+0000 7ff42de81640 1 -- 172.21.15.1:0/3803053561 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4280fc920 con 0x7ff4280fb570 2024-06-18T02:39:39.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:39.915+0000 7ff40ffff640 1 -- 172.21.15.1:0/3803053561 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7ff418003c30 con 0x7ff4280fb570 2024-06-18T02:39:39.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:39.916+0000 7ff40ffff640 1 --2- 172.21.15.1:0/3803053561 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7ff410065430 0x7ff4100678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:39:39.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:39.916+0000 7ff40ffff640 1 -- 172.21.15.1:0/3803053561 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7ff418082480 con 0x7ff4280fb570 2024-06-18T02:39:39.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:39.916+0000 7ff426ffd640 1 --2- 172.21.15.1:0/3803053561 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7ff410065430 0x7ff4100678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:39:39.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:39.916+0000 7ff426ffd640 1 --2- 172.21.15.1:0/3803053561 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7ff410065430 0x7ff4100678e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff41c009f80 tx=0x7ff41c035040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:39:39.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:39.920+0000 7ff40ffff640 1 -- 172.21.15.1:0/3803053561 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7ff418050930 con 0x7ff4280fb570 2024-06-18T02:39:40.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:40.128+0000 7ff42de81640 1 -- 172.21.15.1:0/3803053561 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff428072c20 con 0x7ff4280fb570 2024-06-18T02:39:40.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:39:39 smithi001 ceph-mon[29099]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:39:40.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:40.128+0000 7ff40ffff640 1 -- 172.21.15.1:0/3803053561 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7ff418053f50 con 0x7ff4280fb570 2024-06-18T02:39:40.129 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:39:40.129 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-18T02:39:40.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:40.130+0000 7ff42de81640 1 -- 172.21.15.1:0/3803053561 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7ff410065430 msgr2=0x7ff4100678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:39:40.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:40.130+0000 7ff42de81640 1 --2- 172.21.15.1:0/3803053561 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7ff410065430 0x7ff4100678e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff41c009f80 tx=0x7ff41c035040 comp rx=0 tx=0).stop 2024-06-18T02:39:40.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:40.130+0000 7ff42de81640 1 -- 172.21.15.1:0/3803053561 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4280fb570 msgr2=0x7ff42810e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:39:40.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:40.130+0000 7ff42de81640 1 --2- 172.21.15.1:0/3803053561 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4280fb570 0x7ff42810e0d0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7ff41800db00 tx=0x7ff418004e60 comp rx=0 tx=0).stop 2024-06-18T02:39:40.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:40.130+0000 7ff42de81640 1 -- 172.21.15.1:0/3803053561 shutdown_connections 2024-06-18T02:39:40.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:40.130+0000 7ff42de81640 1 --2- 172.21.15.1:0/3803053561 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7ff410065430 0x7ff4100678e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:39:40.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:40.130+0000 7ff42de81640 1 --2- 172.21.15.1:0/3803053561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4280fbe80 0x7ff42810e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:39:40.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:40.130+0000 7ff42de81640 1 --2- 172.21.15.1:0/3803053561 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff4280fb570 0x7ff42810e0d0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:39:40.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:40.130+0000 7ff42de81640 1 -- 172.21.15.1:0/3803053561 >> 172.21.15.1:0/3803053561 conn(0x7ff428072110 msgr2=0x7ff4280ffea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:39:40.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:40.130+0000 7ff42de81640 1 -- 172.21.15.1:0/3803053561 shutdown_connections 2024-06-18T02:39:40.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:40.130+0000 7ff42de81640 1 -- 172.21.15.1:0/3803053561 wait complete. 2024-06-18T02:39:40.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:39:39 smithi028 ceph-mon[35823]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:39:41.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:39:40 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/3803053561' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:39:41.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:39:40 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/3803053561' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:39:42.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:39:41 smithi028 ceph-mon[35823]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:39:42.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:39:41 smithi001 ceph-mon[29099]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:39:43.783 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T02:39:44.060 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:39:44.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:39:43 smithi001 ceph-mon[29099]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:39:44.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:39:43 smithi028 ceph-mon[35823]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:39:45.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:45.635+0000 7f0e1871c640 1 -- 172.21.15.1:0/2340371729 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0e10102c90 msgr2=0x7f0e100fa9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:39:45.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:45.635+0000 7f0e1871c640 1 --2- 172.21.15.1:0/2340371729 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0e10102c90 0x7f0e100fa9e0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f0e000048d0 tx=0x7f0e00009da0 comp rx=0 tx=0).stop 2024-06-18T02:39:45.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:45.635+0000 7f0e1871c640 1 -- 172.21.15.1:0/2340371729 shutdown_connections 2024-06-18T02:39:45.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:45.635+0000 7f0e1871c640 1 --2- 172.21.15.1:0/2340371729 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0e10102c90 0x7f0e100fa9e0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:39:45.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:45.635+0000 7f0e1871c640 1 --2- 172.21.15.1:0/2340371729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e101022f0 0x7f0e101026c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:39:45.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:45.635+0000 7f0e1871c640 1 -- 172.21.15.1:0/2340371729 >> 172.21.15.1:0/2340371729 conn(0x7f0e10070a40 msgr2=0x7f0e10070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:39:45.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:45.635+0000 7f0e1871c640 1 -- 172.21.15.1:0/2340371729 shutdown_connections 2024-06-18T02:39:45.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:45.635+0000 7f0e1871c640 1 -- 172.21.15.1:0/2340371729 wait complete. 2024-06-18T02:39:45.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:45.636+0000 7f0e1871c640 1 Processor -- start 2024-06-18T02:39:45.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:45.636+0000 7f0e1871c640 1 -- start start 2024-06-18T02:39:45.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:45.636+0000 7f0e1871c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e101022f0 0x7f0e101130e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:39:45.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:45.636+0000 7f0e1871c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0e10102c90 0x7f0e10113620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:39:45.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:45.636+0000 7f0e1871c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e10113c20 con 0x7f0e10102c90 2024-06-18T02:39:45.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:45.636+0000 7f0e1871c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e1010d060 con 0x7f0e101022f0 2024-06-18T02:39:45.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:45.636+0000 7f0e15cb7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0e10102c90 0x7f0e10113620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:39:45.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:45.636+0000 7f0e15cb7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0e10102c90 0x7f0e10113620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55504/0 (socket says 172.21.15.1:55504) 2024-06-18T02:39:45.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:45.636+0000 7f0e15cb7640 1 -- 172.21.15.1:0/3846441223 learned_addr learned my addr 172.21.15.1:0/3846441223 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:39:45.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:45.636+0000 7f0e164b8640 1 --2- 172.21.15.1:0/3846441223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e101022f0 0x7f0e101130e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:39:45.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:45.636+0000 7f0e15cb7640 1 -- 172.21.15.1:0/3846441223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e101022f0 msgr2=0x7f0e101130e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:39:45.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:45.636+0000 7f0e15cb7640 1 --2- 172.21.15.1:0/3846441223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e101022f0 0x7f0e101130e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:39:45.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:45.636+0000 7f0e15cb7640 1 -- 172.21.15.1:0/3846441223 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e00005040 con 0x7f0e10102c90 2024-06-18T02:39:45.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:45.637+0000 7f0e15cb7640 1 --2- 172.21.15.1:0/3846441223 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0e10102c90 0x7f0e10113620 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f0e00003080 tx=0x7f0e00009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:39:45.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:45.637+0000 7f0e06ffd640 1 -- 172.21.15.1:0/3846441223 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e00038070 con 0x7f0e10102c90 2024-06-18T02:39:45.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:45.637+0000 7f0e1871c640 1 -- 172.21.15.1:0/3846441223 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e1010d2e0 con 0x7f0e10102c90 2024-06-18T02:39:45.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:45.637+0000 7f0e06ffd640 1 -- 172.21.15.1:0/3846441223 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0e00008ee0 con 0x7f0e10102c90 2024-06-18T02:39:45.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:45.637+0000 7f0e06ffd640 1 -- 172.21.15.1:0/3846441223 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e00033760 con 0x7f0e10102c90 2024-06-18T02:39:45.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:45.637+0000 7f0e1871c640 1 -- 172.21.15.1:0/3846441223 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e1010d740 con 0x7f0e10102c90 2024-06-18T02:39:45.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:45.638+0000 7f0e1871c640 1 -- 172.21.15.1:0/3846441223 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e10065510 con 0x7f0e10102c90 2024-06-18T02:39:45.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:45.638+0000 7f0e06ffd640 1 -- 172.21.15.1:0/3846441223 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7f0e00030020 con 0x7f0e10102c90 2024-06-18T02:39:45.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:45.639+0000 7f0e06ffd640 1 --2- 172.21.15.1:0/3846441223 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f0dfc065430 0x7f0dfc0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:39:45.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:45.639+0000 7f0e06ffd640 1 -- 172.21.15.1:0/3846441223 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f0e000a2070 con 0x7f0e10102c90 2024-06-18T02:39:45.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:45.639+0000 7f0e164b8640 1 --2- 172.21.15.1:0/3846441223 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f0dfc065430 0x7f0dfc0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:39:45.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:45.639+0000 7f0e164b8640 1 --2- 172.21.15.1:0/3846441223 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f0dfc065430 0x7f0dfc0678e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f0e10065260 tx=0x7f0e0c009040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:39:45.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:45.643+0000 7f0e06ffd640 1 -- 172.21.15.1:0/3846441223 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f0e00070520 con 0x7f0e10102c90 2024-06-18T02:39:45.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:45.854+0000 7f0e1871c640 1 -- 172.21.15.1:0/3846441223 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f0e1010e780 con 0x7f0e10102c90 2024-06-18T02:39:45.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:45.855+0000 7f0e06ffd640 1 -- 172.21.15.1:0/3846441223 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f0e00079020 con 0x7f0e10102c90 2024-06-18T02:39:45.855 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:39:45.855 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-18T02:39:45.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:45.857+0000 7f0e1871c640 1 -- 172.21.15.1:0/3846441223 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f0dfc065430 msgr2=0x7f0dfc0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:39:45.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:45.857+0000 7f0e1871c640 1 --2- 172.21.15.1:0/3846441223 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f0dfc065430 0x7f0dfc0678e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f0e10065260 tx=0x7f0e0c009040 comp rx=0 tx=0).stop 2024-06-18T02:39:45.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:45.857+0000 7f0e1871c640 1 -- 172.21.15.1:0/3846441223 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0e10102c90 msgr2=0x7f0e10113620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:39:45.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:45.857+0000 7f0e1871c640 1 --2- 172.21.15.1:0/3846441223 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0e10102c90 0x7f0e10113620 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f0e00003080 tx=0x7f0e00009da0 comp rx=0 tx=0).stop 2024-06-18T02:39:45.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:45.857+0000 7f0e1871c640 1 -- 172.21.15.1:0/3846441223 shutdown_connections 2024-06-18T02:39:45.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:45.857+0000 7f0e1871c640 1 --2- 172.21.15.1:0/3846441223 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f0dfc065430 0x7f0dfc0678e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:39:45.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:45.857+0000 7f0e1871c640 1 --2- 172.21.15.1:0/3846441223 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0e10102c90 0x7f0e10113620 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:39:45.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:45.857+0000 7f0e1871c640 1 --2- 172.21.15.1:0/3846441223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e101022f0 0x7f0e101130e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:39:45.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:45.857+0000 7f0e1871c640 1 -- 172.21.15.1:0/3846441223 >> 172.21.15.1:0/3846441223 conn(0x7f0e10070a40 msgr2=0x7f0e10105090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:39:45.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:45.857+0000 7f0e1871c640 1 -- 172.21.15.1:0/3846441223 shutdown_connections 2024-06-18T02:39:45.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:45.857+0000 7f0e1871c640 1 -- 172.21.15.1:0/3846441223 wait complete. 2024-06-18T02:39:46.127 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:39:45 smithi001 ceph-mon[29099]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:39:46.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:39:45 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/3846441223' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:39:46.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:39:45 smithi028 ceph-mon[35823]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:39:46.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:39:45 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/3846441223' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:39:48.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:39:47 smithi001 ceph-mon[29099]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:39:48.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:39:47 smithi028 ceph-mon[35823]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:39:49.504 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T02:39:49.782 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:39:50.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:39:49 smithi028 ceph-mon[35823]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:39:50.247 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:39:49 smithi001 ceph-mon[29099]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:39:51.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:51.409+0000 7f48349bb640 1 -- 172.21.15.1:0/3286161881 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f48301022d0 msgr2=0x7f48301026a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:39:51.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:51.409+0000 7f48349bb640 1 --2- 172.21.15.1:0/3286161881 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f48301022d0 0x7f48301026a0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f48200048d0 tx=0x7f4820009da0 comp rx=0 tx=0).stop 2024-06-18T02:39:51.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:51.409+0000 7f48349bb640 1 -- 172.21.15.1:0/3286161881 shutdown_connections 2024-06-18T02:39:51.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:51.409+0000 7f48349bb640 1 --2- 172.21.15.1:0/3286161881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4830102c70 0x7f48300fc300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:39:51.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:51.409+0000 7f48349bb640 1 --2- 172.21.15.1:0/3286161881 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f48301022d0 0x7f48301026a0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:39:51.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:51.409+0000 7f48349bb640 1 -- 172.21.15.1:0/3286161881 >> 172.21.15.1:0/3286161881 conn(0x7f48300f7e60 msgr2=0x7f48300fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:39:51.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:51.409+0000 7f48349bb640 1 -- 172.21.15.1:0/3286161881 shutdown_connections 2024-06-18T02:39:51.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:51.409+0000 7f48349bb640 1 -- 172.21.15.1:0/3286161881 wait complete. 2024-06-18T02:39:51.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:51.410+0000 7f48349bb640 1 Processor -- start 2024-06-18T02:39:51.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:51.410+0000 7f48349bb640 1 -- start start 2024-06-18T02:39:51.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:51.410+0000 7f48349bb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48301022d0 0x7f4830113060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:39:51.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:51.410+0000 7f48349bb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4830102c70 0x7f48301135a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:39:51.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:51.410+0000 7f48349bb640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4830113c30 con 0x7f4830102c70 2024-06-18T02:39:51.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:51.410+0000 7f48349bb640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f483010cfe0 con 0x7f48301022d0 2024-06-18T02:39:51.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:51.410+0000 7f482dd9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4830102c70 0x7f48301135a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:39:51.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:51.410+0000 7f482dd9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4830102c70 0x7f48301135a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55522/0 (socket says 172.21.15.1:55522) 2024-06-18T02:39:51.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:51.410+0000 7f482dd9b640 1 -- 172.21.15.1:0/1264447357 learned_addr learned my addr 172.21.15.1:0/1264447357 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:39:51.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:51.410+0000 7f482e59c640 1 --2- 172.21.15.1:0/1264447357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48301022d0 0x7f4830113060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:39:51.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:51.410+0000 7f482dd9b640 1 -- 172.21.15.1:0/1264447357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48301022d0 msgr2=0x7f4830113060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:39:51.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:51.410+0000 7f482dd9b640 1 --2- 172.21.15.1:0/1264447357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48301022d0 0x7f4830113060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:39:51.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:51.410+0000 7f482dd9b640 1 -- 172.21.15.1:0/1264447357 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4820005040 con 0x7f4830102c70 2024-06-18T02:39:51.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:51.411+0000 7f482dd9b640 1 --2- 172.21.15.1:0/1264447357 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4830102c70 0x7f48301135a0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f481800b790 tx=0x7f48180091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:39:51.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:51.411+0000 7f4826ffd640 1 -- 172.21.15.1:0/1264447357 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f481800c700 con 0x7f4830102c70 2024-06-18T02:39:51.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:51.411+0000 7f4826ffd640 1 -- 172.21.15.1:0/1264447357 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f481800cd30 con 0x7f4830102c70 2024-06-18T02:39:51.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:51.411+0000 7f48349bb640 1 -- 172.21.15.1:0/1264447357 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f483010d2c0 con 0x7f4830102c70 2024-06-18T02:39:51.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:51.411+0000 7f4826ffd640 1 -- 172.21.15.1:0/1264447357 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4818013370 con 0x7f4830102c70 2024-06-18T02:39:51.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:51.411+0000 7f48349bb640 1 -- 172.21.15.1:0/1264447357 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f483010d7e0 con 0x7f4830102c70 2024-06-18T02:39:51.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:51.412+0000 7f48349bb640 1 -- 172.21.15.1:0/1264447357 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48300fda40 con 0x7f4830102c70 2024-06-18T02:39:51.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:51.412+0000 7f4826ffd640 1 -- 172.21.15.1:0/1264447357 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7f4818012070 con 0x7f4830102c70 2024-06-18T02:39:51.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:51.413+0000 7f4826ffd640 1 --2- 172.21.15.1:0/1264447357 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f481c065430 0x7f481c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:39:51.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:51.413+0000 7f4826ffd640 1 -- 172.21.15.1:0/1264447357 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f4818081dc0 con 0x7f4830102c70 2024-06-18T02:39:51.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:51.413+0000 7f482e59c640 1 --2- 172.21.15.1:0/1264447357 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f481c065430 0x7f481c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:39:51.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:51.413+0000 7f482e59c640 1 --2- 172.21.15.1:0/1264447357 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f481c065430 0x7f481c0678e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f4820009f80 tx=0x7f4820035040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:39:51.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:51.417+0000 7f4826ffd640 1 -- 172.21.15.1:0/1264447357 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f4818050270 con 0x7f4830102c70 2024-06-18T02:39:51.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:51.629+0000 7f48349bb640 1 -- 172.21.15.1:0/1264447357 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f48301094c0 con 0x7f4830102c70 2024-06-18T02:39:51.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:51.629+0000 7f4826ffd640 1 -- 172.21.15.1:0/1264447357 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f4818053890 con 0x7f4830102c70 2024-06-18T02:39:51.630 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:39:51.630 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-18T02:39:51.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:51.632+0000 7f48349bb640 1 -- 172.21.15.1:0/1264447357 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f481c065430 msgr2=0x7f481c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:39:51.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:51.632+0000 7f48349bb640 1 --2- 172.21.15.1:0/1264447357 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f481c065430 0x7f481c0678e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f4820009f80 tx=0x7f4820035040 comp rx=0 tx=0).stop 2024-06-18T02:39:51.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:51.632+0000 7f48349bb640 1 -- 172.21.15.1:0/1264447357 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4830102c70 msgr2=0x7f48301135a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:39:51.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:51.632+0000 7f48349bb640 1 --2- 172.21.15.1:0/1264447357 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4830102c70 0x7f48301135a0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f481800b790 tx=0x7f48180091d0 comp rx=0 tx=0).stop 2024-06-18T02:39:51.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:51.632+0000 7f48349bb640 1 -- 172.21.15.1:0/1264447357 shutdown_connections 2024-06-18T02:39:51.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:51.632+0000 7f48349bb640 1 --2- 172.21.15.1:0/1264447357 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f481c065430 0x7f481c0678e0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:39:51.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:51.632+0000 7f48349bb640 1 --2- 172.21.15.1:0/1264447357 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4830102c70 0x7f48301135a0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:39:51.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:51.632+0000 7f48349bb640 1 --2- 172.21.15.1:0/1264447357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48301022d0 0x7f4830113060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:39:51.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:51.632+0000 7f48349bb640 1 -- 172.21.15.1:0/1264447357 >> 172.21.15.1:0/1264447357 conn(0x7f48300f7e60 msgr2=0x7f48300f8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:39:51.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:51.632+0000 7f48349bb640 1 -- 172.21.15.1:0/1264447357 shutdown_connections 2024-06-18T02:39:51.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:51.632+0000 7f48349bb640 1 -- 172.21.15.1:0/1264447357 wait complete. 2024-06-18T02:39:52.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:39:51 smithi028 ceph-mon[35823]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:39:52.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:39:51 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:39:52.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:39:51 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/1264447357' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:39:52.246 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:39:51 smithi001 ceph-mon[29099]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:39:52.246 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:39:51 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:39:52.246 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:39:51 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/1264447357' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:39:54.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:39:53 smithi028 ceph-mon[35823]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:39:54.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:39:53 smithi001 ceph-mon[29099]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:39:55.274 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T02:39:55.550 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:39:56.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:39:55 smithi028 ceph-mon[35823]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:39:56.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:39:55 smithi001 ceph-mon[29099]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:39:57.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:57.102+0000 7f957d3a8640 1 -- 172.21.15.1:0/2877514449 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95781022f0 msgr2=0x7f95781026c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:39:57.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:57.102+0000 7f957d3a8640 1 --2- 172.21.15.1:0/2877514449 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95781022f0 0x7f95781026c0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f95680048d0 tx=0x7f9568009da0 comp rx=0 tx=0).stop 2024-06-18T02:39:57.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:57.102+0000 7f957d3a8640 1 -- 172.21.15.1:0/2877514449 shutdown_connections 2024-06-18T02:39:57.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:57.102+0000 7f957d3a8640 1 --2- 172.21.15.1:0/2877514449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9578102c90 0x7f95780fa9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:39:57.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:57.102+0000 7f957d3a8640 1 --2- 172.21.15.1:0/2877514449 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95781022f0 0x7f95781026c0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:39:57.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:57.102+0000 7f957d3a8640 1 -- 172.21.15.1:0/2877514449 >> 172.21.15.1:0/2877514449 conn(0x7f9578070a40 msgr2=0x7f9578070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:39:57.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:57.102+0000 7f957d3a8640 1 -- 172.21.15.1:0/2877514449 shutdown_connections 2024-06-18T02:39:57.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:57.102+0000 7f957d3a8640 1 -- 172.21.15.1:0/2877514449 wait complete. 2024-06-18T02:39:57.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:57.103+0000 7f957d3a8640 1 Processor -- start 2024-06-18T02:39:57.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:57.103+0000 7f957d3a8640 1 -- start start 2024-06-18T02:39:57.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:57.103+0000 7f957d3a8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95781022f0 0x7f9578113170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:39:57.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:57.103+0000 7f957d3a8640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9578102c90 0x7f95781136b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:39:57.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:57.103+0000 7f957d3a8640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9578113d40 con 0x7f9578102c90 2024-06-18T02:39:57.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:57.103+0000 7f957d3a8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f957810d0f0 con 0x7f95781022f0 2024-06-18T02:39:57.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:57.103+0000 7f95767fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9578102c90 0x7f95781136b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:39:57.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:57.103+0000 7f95767fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9578102c90 0x7f95781136b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39140/0 (socket says 172.21.15.1:39140) 2024-06-18T02:39:57.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:57.103+0000 7f95767fc640 1 -- 172.21.15.1:0/2722629920 learned_addr learned my addr 172.21.15.1:0/2722629920 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:39:57.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:57.103+0000 7f9576ffd640 1 --2- 172.21.15.1:0/2722629920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95781022f0 0x7f9578113170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:39:57.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:57.103+0000 7f95767fc640 1 -- 172.21.15.1:0/2722629920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95781022f0 msgr2=0x7f9578113170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:39:57.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:57.103+0000 7f95767fc640 1 --2- 172.21.15.1:0/2722629920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95781022f0 0x7f9578113170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:39:57.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:57.103+0000 7f95767fc640 1 -- 172.21.15.1:0/2722629920 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9568005040 con 0x7f9578102c90 2024-06-18T02:39:57.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:57.103+0000 7f9576ffd640 1 --2- 172.21.15.1:0/2722629920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95781022f0 0x7f9578113170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-18T02:39:57.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:57.104+0000 7f95767fc640 1 --2- 172.21.15.1:0/2722629920 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9578102c90 0x7f95781136b0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f956000b540 tx=0x7f95600091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:39:57.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:57.104+0000 7f956f7fe640 1 -- 172.21.15.1:0/2722629920 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f956000f6f0 con 0x7f9578102c90 2024-06-18T02:39:57.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:57.104+0000 7f956f7fe640 1 -- 172.21.15.1:0/2722629920 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f956000fd20 con 0x7f9578102c90 2024-06-18T02:39:57.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:57.104+0000 7f957d3a8640 1 -- 172.21.15.1:0/2722629920 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f957810d3d0 con 0x7f9578102c90 2024-06-18T02:39:57.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:57.104+0000 7f956f7fe640 1 -- 172.21.15.1:0/2722629920 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f95600103a0 con 0x7f9578102c90 2024-06-18T02:39:57.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:57.104+0000 7f957d3a8640 1 -- 172.21.15.1:0/2722629920 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f957810d8f0 con 0x7f9578102c90 2024-06-18T02:39:57.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:57.105+0000 7f957d3a8640 1 -- 172.21.15.1:0/2722629920 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9578065510 con 0x7f9578102c90 2024-06-18T02:39:57.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:57.105+0000 7f956f7fe640 1 -- 172.21.15.1:0/2722629920 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7f956000f850 con 0x7f9578102c90 2024-06-18T02:39:57.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:57.106+0000 7f956f7fe640 1 --2- 172.21.15.1:0/2722629920 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f9564065430 0x7f95640678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:39:57.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:57.106+0000 7f956f7fe640 1 -- 172.21.15.1:0/2722629920 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f9560082350 con 0x7f9578102c90 2024-06-18T02:39:57.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:57.106+0000 7f9576ffd640 1 --2- 172.21.15.1:0/2722629920 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f9564065430 0x7f95640678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:39:57.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:57.106+0000 7f9576ffd640 1 --2- 172.21.15.1:0/2722629920 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f9564065430 0x7f95640678e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f95680043d0 tx=0x7f9568009da0 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:39:57.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:57.110+0000 7f956f7fe640 1 -- 172.21.15.1:0/2722629920 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f9560050920 con 0x7f9578102c90 2024-06-18T02:39:57.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:57.318+0000 7f957d3a8640 1 -- 172.21.15.1:0/2722629920 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f9578109550 con 0x7f9578102c90 2024-06-18T02:39:57.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:57.318+0000 7f956f7fe640 1 -- 172.21.15.1:0/2722629920 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f9560053f40 con 0x7f9578102c90 2024-06-18T02:39:57.320 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:39:57.320 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-18T02:39:57.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:57.320+0000 7f957d3a8640 1 -- 172.21.15.1:0/2722629920 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f9564065430 msgr2=0x7f95640678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:39:57.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:57.320+0000 7f957d3a8640 1 --2- 172.21.15.1:0/2722629920 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f9564065430 0x7f95640678e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f95680043d0 tx=0x7f9568009da0 comp rx=0 tx=0).stop 2024-06-18T02:39:57.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:57.320+0000 7f957d3a8640 1 -- 172.21.15.1:0/2722629920 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9578102c90 msgr2=0x7f95781136b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:39:57.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:57.320+0000 7f957d3a8640 1 --2- 172.21.15.1:0/2722629920 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9578102c90 0x7f95781136b0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f956000b540 tx=0x7f95600091a0 comp rx=0 tx=0).stop 2024-06-18T02:39:57.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:57.321+0000 7f957d3a8640 1 -- 172.21.15.1:0/2722629920 shutdown_connections 2024-06-18T02:39:57.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:57.321+0000 7f957d3a8640 1 --2- 172.21.15.1:0/2722629920 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f9564065430 0x7f95640678e0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:39:57.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:57.321+0000 7f957d3a8640 1 --2- 172.21.15.1:0/2722629920 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9578102c90 0x7f95781136b0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:39:57.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:57.321+0000 7f957d3a8640 1 --2- 172.21.15.1:0/2722629920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95781022f0 0x7f9578113170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:39:57.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:57.321+0000 7f957d3a8640 1 -- 172.21.15.1:0/2722629920 >> 172.21.15.1:0/2722629920 conn(0x7f9578070a40 msgr2=0x7f95781050b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:39:57.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:57.321+0000 7f957d3a8640 1 -- 172.21.15.1:0/2722629920 shutdown_connections 2024-06-18T02:39:57.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:39:57.321+0000 7f957d3a8640 1 -- 172.21.15.1:0/2722629920 wait complete. 2024-06-18T02:39:58.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:39:57 smithi028 ceph-mon[35823]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:39:58.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:39:57 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/2722629920' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:39:58.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:39:57 smithi001 ceph-mon[29099]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:39:58.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:39:57 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/2722629920' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:40:00.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:39:59 smithi028 ceph-mon[35823]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:40:00.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:39:59 smithi001 ceph-mon[29099]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:40:00.973 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T02:40:01.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:40:00 smithi028 ceph-mon[35823]: overall HEALTH_WARN 3 mgr modules have recently crashed 2024-06-18T02:40:01.251 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:40:01.374 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:40:00 smithi001 ceph-mon[29099]: overall HEALTH_WARN 3 mgr modules have recently crashed 2024-06-18T02:40:02.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:40:01 smithi001 ceph-mon[29099]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:40:02.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:40:01 smithi028 ceph-mon[35823]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:40:02.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:02.982+0000 7f586cba3640 1 -- 172.21.15.1:0/3232946687 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f58681022e0 msgr2=0x7f58681026b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:40:02.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:02.982+0000 7f586cba3640 1 --2- 172.21.15.1:0/3232946687 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f58681022e0 0x7f58681026b0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f58580048d0 tx=0x7f5858009da0 comp rx=0 tx=0).stop 2024-06-18T02:40:02.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:02.982+0000 7f586cba3640 1 -- 172.21.15.1:0/3232946687 shutdown_connections 2024-06-18T02:40:02.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:02.982+0000 7f586cba3640 1 --2- 172.21.15.1:0/3232946687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5868102c80 0x7f58680fc310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:40:02.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:02.982+0000 7f586cba3640 1 --2- 172.21.15.1:0/3232946687 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f58681022e0 0x7f58681026b0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:40:02.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:02.982+0000 7f586cba3640 1 -- 172.21.15.1:0/3232946687 >> 172.21.15.1:0/3232946687 conn(0x7f58680f7e70 msgr2=0x7f58680fa280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:40:02.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:02.982+0000 7f586cba3640 1 -- 172.21.15.1:0/3232946687 shutdown_connections 2024-06-18T02:40:02.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:02.982+0000 7f586cba3640 1 -- 172.21.15.1:0/3232946687 wait complete. 2024-06-18T02:40:02.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:02.982+0000 7f586cba3640 1 Processor -- start 2024-06-18T02:40:02.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:02.982+0000 7f586cba3640 1 -- start start 2024-06-18T02:40:02.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:02.983+0000 7f586cba3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58681022e0 0x7f586806f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:40:02.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:02.983+0000 7f586cba3640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5868102c80 0x7f586806fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:40:02.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:02.983+0000 7f586cba3640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5868070280 con 0x7f5868102c80 2024-06-18T02:40:02.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:02.983+0000 7f586cba3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58680703f0 con 0x7f58681022e0 2024-06-18T02:40:02.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:02.983+0000 7f5865d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5868102c80 0x7f586806fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:40:02.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:02.983+0000 7f586659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58681022e0 0x7f586806f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:40:02.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:02.983+0000 7f5865d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5868102c80 0x7f586806fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39154/0 (socket says 172.21.15.1:39154) 2024-06-18T02:40:02.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:02.983+0000 7f5865d9b640 1 -- 172.21.15.1:0/1628526401 learned_addr learned my addr 172.21.15.1:0/1628526401 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:40:02.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:02.983+0000 7f5865d9b640 1 -- 172.21.15.1:0/1628526401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58681022e0 msgr2=0x7f586806f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:40:02.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:02.983+0000 7f5865d9b640 1 --2- 172.21.15.1:0/1628526401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58681022e0 0x7f586806f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:40:02.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:02.983+0000 7f5865d9b640 1 -- 172.21.15.1:0/1628526401 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5858005040 con 0x7f5868102c80 2024-06-18T02:40:02.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:02.983+0000 7f5865d9b640 1 --2- 172.21.15.1:0/1628526401 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5868102c80 0x7f586806fcb0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f585c00b790 tx=0x7f585c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:40:02.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:02.983+0000 7f5856ffd640 1 -- 172.21.15.1:0/1628526401 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f585c00c700 con 0x7f5868102c80 2024-06-18T02:40:02.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:02.983+0000 7f5856ffd640 1 -- 172.21.15.1:0/1628526401 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f585c00cd30 con 0x7f5868102c80 2024-06-18T02:40:02.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:02.983+0000 7f586cba3640 1 -- 172.21.15.1:0/1628526401 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5868119370 con 0x7f5868102c80 2024-06-18T02:40:02.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:02.983+0000 7f5856ffd640 1 -- 172.21.15.1:0/1628526401 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f585c013370 con 0x7f5868102c80 2024-06-18T02:40:02.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:02.983+0000 7f586cba3640 1 -- 172.21.15.1:0/1628526401 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58681197b0 con 0x7f5868102c80 2024-06-18T02:40:02.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:02.984+0000 7f586cba3640 1 -- 172.21.15.1:0/1628526401 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58680fda50 con 0x7f5868102c80 2024-06-18T02:40:02.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:02.985+0000 7f5856ffd640 1 -- 172.21.15.1:0/1628526401 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7f585c012070 con 0x7f5868102c80 2024-06-18T02:40:02.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:02.985+0000 7f5856ffd640 1 --2- 172.21.15.1:0/1628526401 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f5850065430 0x7f58500678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:40:02.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:02.985+0000 7f5856ffd640 1 -- 172.21.15.1:0/1628526401 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f585c081df0 con 0x7f5868102c80 2024-06-18T02:40:02.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:02.986+0000 7f586659c640 1 --2- 172.21.15.1:0/1628526401 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f5850065430 0x7f58500678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:40:02.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:02.986+0000 7f586659c640 1 --2- 172.21.15.1:0/1628526401 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f5850065430 0x7f58500678e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5858032430 tx=0x7f5858035040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:40:02.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:02.990+0000 7f5856ffd640 1 -- 172.21.15.1:0/1628526401 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f585c0502a0 con 0x7f5868102c80 2024-06-18T02:40:03.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:03.197+0000 7f586cba3640 1 -- 172.21.15.1:0/1628526401 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f58681094d0 con 0x7f5868102c80 2024-06-18T02:40:03.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:03.197+0000 7f5856ffd640 1 -- 172.21.15.1:0/1628526401 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f585c0538c0 con 0x7f5868102c80 2024-06-18T02:40:03.198 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:40:03.198 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-18T02:40:03.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:03.199+0000 7f586cba3640 1 -- 172.21.15.1:0/1628526401 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f5850065430 msgr2=0x7f58500678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:40:03.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:03.199+0000 7f586cba3640 1 --2- 172.21.15.1:0/1628526401 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f5850065430 0x7f58500678e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5858032430 tx=0x7f5858035040 comp rx=0 tx=0).stop 2024-06-18T02:40:03.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:03.200+0000 7f586cba3640 1 -- 172.21.15.1:0/1628526401 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5868102c80 msgr2=0x7f586806fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:40:03.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:03.200+0000 7f586cba3640 1 --2- 172.21.15.1:0/1628526401 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5868102c80 0x7f586806fcb0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f585c00b790 tx=0x7f585c0091d0 comp rx=0 tx=0).stop 2024-06-18T02:40:03.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:03.200+0000 7f586cba3640 1 -- 172.21.15.1:0/1628526401 shutdown_connections 2024-06-18T02:40:03.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:03.200+0000 7f586cba3640 1 --2- 172.21.15.1:0/1628526401 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f5850065430 0x7f58500678e0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:40:03.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:03.200+0000 7f586cba3640 1 --2- 172.21.15.1:0/1628526401 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5868102c80 0x7f586806fcb0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:40:03.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:03.200+0000 7f586cba3640 1 --2- 172.21.15.1:0/1628526401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58681022e0 0x7f586806f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:40:03.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:03.200+0000 7f586cba3640 1 -- 172.21.15.1:0/1628526401 >> 172.21.15.1:0/1628526401 conn(0x7f58680f7e70 msgr2=0x7f58680f8de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:40:03.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:03.200+0000 7f586cba3640 1 -- 172.21.15.1:0/1628526401 shutdown_connections 2024-06-18T02:40:03.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:03.200+0000 7f586cba3640 1 -- 172.21.15.1:0/1628526401 wait complete. 2024-06-18T02:40:04.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:40:03 smithi001 ceph-mon[29099]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:40:04.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:40:03 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/1628526401' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:40:04.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:40:03 smithi028 ceph-mon[35823]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:40:04.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:40:03 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/1628526401' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:40:06.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:40:05 smithi028 ceph-mon[35823]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:40:06.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:40:05 smithi001 ceph-mon[29099]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:40:06.850 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T02:40:07.127 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:40:07.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:40:06 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:40:07.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:40:06 smithi028 ceph-mon[35823]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:40:07.249 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:40:06 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:40:07.250 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:40:06 smithi001 ceph-mon[29099]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:40:08.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:08.830+0000 7f4ab8c98640 1 -- 172.21.15.1:0/1699520490 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ab41022f0 msgr2=0x7f4ab41026c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:40:08.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:08.830+0000 7f4ab8c98640 1 --2- 172.21.15.1:0/1699520490 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ab41022f0 0x7f4ab41026c0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f4aa40048d0 tx=0x7f4aa4009da0 comp rx=0 tx=0).stop 2024-06-18T02:40:08.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:08.830+0000 7f4ab8c98640 1 -- 172.21.15.1:0/1699520490 shutdown_connections 2024-06-18T02:40:08.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:08.830+0000 7f4ab8c98640 1 --2- 172.21.15.1:0/1699520490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ab4102c90 0x7f4ab40fc320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:40:08.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:08.830+0000 7f4ab8c98640 1 --2- 172.21.15.1:0/1699520490 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ab41022f0 0x7f4ab41026c0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:40:08.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:08.830+0000 7f4ab8c98640 1 -- 172.21.15.1:0/1699520490 >> 172.21.15.1:0/1699520490 conn(0x7f4ab40f7f00 msgr2=0x7f4ab40fa310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:40:08.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:08.830+0000 7f4ab8c98640 1 -- 172.21.15.1:0/1699520490 shutdown_connections 2024-06-18T02:40:08.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:08.830+0000 7f4ab8c98640 1 -- 172.21.15.1:0/1699520490 wait complete. 2024-06-18T02:40:08.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:08.831+0000 7f4ab8c98640 1 Processor -- start 2024-06-18T02:40:08.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:08.831+0000 7f4ab8c98640 1 -- start start 2024-06-18T02:40:08.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:08.831+0000 7f4ab8c98640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ab41022f0 0x7f4ab4113040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:40:08.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:08.831+0000 7f4ab8c98640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ab4102c90 0x7f4ab4113580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:40:08.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:08.831+0000 7f4ab8c98640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ab4113c10 con 0x7f4ab41022f0 2024-06-18T02:40:08.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:08.831+0000 7f4ab8c98640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ab410d000 con 0x7f4ab4102c90 2024-06-18T02:40:08.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:08.831+0000 7f4ab259c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ab41022f0 0x7f4ab4113040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:40:08.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:08.831+0000 7f4ab259c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ab41022f0 0x7f4ab4113040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41012/0 (socket says 172.21.15.1:41012) 2024-06-18T02:40:08.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:08.831+0000 7f4ab259c640 1 -- 172.21.15.1:0/1262228684 learned_addr learned my addr 172.21.15.1:0/1262228684 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:40:08.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:08.831+0000 7f4aabfff640 1 --2- 172.21.15.1:0/1262228684 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ab4102c90 0x7f4ab4113580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:40:08.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:08.831+0000 7f4ab259c640 1 -- 172.21.15.1:0/1262228684 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ab4102c90 msgr2=0x7f4ab4113580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:40:08.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:08.831+0000 7f4ab259c640 1 --2- 172.21.15.1:0/1262228684 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ab4102c90 0x7f4ab4113580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:40:08.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:08.832+0000 7f4ab259c640 1 -- 172.21.15.1:0/1262228684 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aa4005040 con 0x7f4ab41022f0 2024-06-18T02:40:08.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:08.832+0000 7f4ab259c640 1 --2- 172.21.15.1:0/1262228684 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ab41022f0 0x7f4ab4113040 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f4aa4003080 tx=0x7f4aa4009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:40:08.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:08.832+0000 7f4aaaffd640 1 -- 172.21.15.1:0/1262228684 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4aa4038070 con 0x7f4ab41022f0 2024-06-18T02:40:08.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:08.832+0000 7f4aaaffd640 1 -- 172.21.15.1:0/1262228684 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4aa4008ee0 con 0x7f4ab41022f0 2024-06-18T02:40:08.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:08.832+0000 7f4ab8c98640 1 -- 172.21.15.1:0/1262228684 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ab410d280 con 0x7f4ab41022f0 2024-06-18T02:40:08.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:08.832+0000 7f4aaaffd640 1 -- 172.21.15.1:0/1262228684 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4aa403d870 con 0x7f4ab41022f0 2024-06-18T02:40:08.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:08.832+0000 7f4ab8c98640 1 -- 172.21.15.1:0/1262228684 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ab410d740 con 0x7f4ab41022f0 2024-06-18T02:40:08.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:08.833+0000 7f4ab8c98640 1 -- 172.21.15.1:0/1262228684 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ab40fda60 con 0x7f4ab41022f0 2024-06-18T02:40:08.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:08.834+0000 7f4aaaffd640 1 -- 172.21.15.1:0/1262228684 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7f4aa4030020 con 0x7f4ab41022f0 2024-06-18T02:40:08.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:08.834+0000 7f4aaaffd640 1 --2- 172.21.15.1:0/1262228684 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f4aa0065430 0x7f4aa00678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:40:08.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:08.834+0000 7f4aaaffd640 1 -- 172.21.15.1:0/1262228684 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f4aa40a2280 con 0x7f4ab41022f0 2024-06-18T02:40:08.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:08.834+0000 7f4aabfff640 1 --2- 172.21.15.1:0/1262228684 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f4aa0065430 0x7f4aa00678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:40:08.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:08.835+0000 7f4aabfff640 1 --2- 172.21.15.1:0/1262228684 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f4aa0065430 0x7f4aa00678e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f4ab410e940 tx=0x7f4a9c008040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:40:08.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:08.838+0000 7f4aaaffd640 1 -- 172.21.15.1:0/1262228684 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f4aa40707b0 con 0x7f4ab41022f0 2024-06-18T02:40:09.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:09.050+0000 7f4ab8c98640 1 -- 172.21.15.1:0/1262228684 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4ab40fbb00 con 0x7f4ab41022f0 2024-06-18T02:40:09.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:09.051+0000 7f4aaaffd640 1 -- 172.21.15.1:0/1262228684 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f4aa4073dd0 con 0x7f4ab41022f0 2024-06-18T02:40:09.052 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:40:09.052 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-18T02:40:09.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:09.053+0000 7f4ab8c98640 1 -- 172.21.15.1:0/1262228684 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f4aa0065430 msgr2=0x7f4aa00678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:40:09.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:09.053+0000 7f4ab8c98640 1 --2- 172.21.15.1:0/1262228684 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f4aa0065430 0x7f4aa00678e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f4ab410e940 tx=0x7f4a9c008040 comp rx=0 tx=0).stop 2024-06-18T02:40:09.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:09.053+0000 7f4ab8c98640 1 -- 172.21.15.1:0/1262228684 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ab41022f0 msgr2=0x7f4ab4113040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:40:09.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:09.053+0000 7f4ab8c98640 1 --2- 172.21.15.1:0/1262228684 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ab41022f0 0x7f4ab4113040 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f4aa4003080 tx=0x7f4aa4009da0 comp rx=0 tx=0).stop 2024-06-18T02:40:09.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:09.053+0000 7f4ab8c98640 1 -- 172.21.15.1:0/1262228684 shutdown_connections 2024-06-18T02:40:09.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:09.053+0000 7f4ab8c98640 1 --2- 172.21.15.1:0/1262228684 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f4aa0065430 0x7f4aa00678e0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:40:09.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:09.053+0000 7f4ab8c98640 1 --2- 172.21.15.1:0/1262228684 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ab4102c90 0x7f4ab4113580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:40:09.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:09.053+0000 7f4ab8c98640 1 --2- 172.21.15.1:0/1262228684 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ab41022f0 0x7f4ab4113040 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:40:09.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:09.053+0000 7f4ab8c98640 1 -- 172.21.15.1:0/1262228684 >> 172.21.15.1:0/1262228684 conn(0x7f4ab40f7f00 msgr2=0x7f4ab40f8e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:40:09.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:09.053+0000 7f4ab8c98640 1 -- 172.21.15.1:0/1262228684 shutdown_connections 2024-06-18T02:40:09.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:09.053+0000 7f4ab8c98640 1 -- 172.21.15.1:0/1262228684 wait complete. 2024-06-18T02:40:09.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:40:09 smithi001 ceph-mon[29099]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:40:09.378 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:40:09 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/1262228684' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:40:09.487 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:40:09 smithi028 ceph-mon[35823]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:40:09.488 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:40:09 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/1262228684' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:40:11.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:40:11 smithi028 ceph-mon[35823]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:40:12.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:40:11 smithi001 ceph-mon[29099]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:40:12.680 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T02:40:12.956 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:40:13.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:40:13 smithi028 ceph-mon[35823]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:40:14.059 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:40:13 smithi001 ceph-mon[29099]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:40:14.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:14.558+0000 7fce08807640 1 -- 172.21.15.1:0/3587057747 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce04101b50 msgr2=0x7fce040fb1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:40:14.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:14.558+0000 7fce08807640 1 --2- 172.21.15.1:0/3587057747 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce04101b50 0x7fce040fb1e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fcdf80048d0 tx=0x7fcdf8009da0 comp rx=0 tx=0).stop 2024-06-18T02:40:14.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:14.558+0000 7fce08807640 1 -- 172.21.15.1:0/3587057747 shutdown_connections 2024-06-18T02:40:14.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:14.558+0000 7fce08807640 1 --2- 172.21.15.1:0/3587057747 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce04101b50 0x7fce040fb1e0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:40:14.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:14.558+0000 7fce08807640 1 --2- 172.21.15.1:0/3587057747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce041011b0 0x7fce04101580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:40:14.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:14.558+0000 7fce08807640 1 -- 172.21.15.1:0/3587057747 >> 172.21.15.1:0/3587057747 conn(0x7fce040f6d60 msgr2=0x7fce040f9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:40:14.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:14.558+0000 7fce08807640 1 -- 172.21.15.1:0/3587057747 shutdown_connections 2024-06-18T02:40:14.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:14.558+0000 7fce08807640 1 -- 172.21.15.1:0/3587057747 wait complete. 2024-06-18T02:40:14.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:14.558+0000 7fce08807640 1 Processor -- start 2024-06-18T02:40:14.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:14.559+0000 7fce08807640 1 -- start start 2024-06-18T02:40:14.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:14.559+0000 7fce08807640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce041011b0 0x7fce04102030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:40:14.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:14.559+0000 7fce08807640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce04101b50 0x7fce04102570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:40:14.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:14.559+0000 7fce08807640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce04102b40 con 0x7fce041011b0 2024-06-18T02:40:14.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:14.559+0000 7fce08807640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce04102cb0 con 0x7fce04101b50 2024-06-18T02:40:14.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:14.559+0000 7fce0259c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce041011b0 0x7fce04102030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:40:14.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:14.559+0000 7fce0259c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce041011b0 0x7fce04102030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47952/0 (socket says 172.21.15.1:47952) 2024-06-18T02:40:14.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:14.559+0000 7fce0259c640 1 -- 172.21.15.1:0/210607512 learned_addr learned my addr 172.21.15.1:0/210607512 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:40:14.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:14.559+0000 7fce01d9b640 1 --2- 172.21.15.1:0/210607512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce04101b50 0x7fce04102570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:40:14.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:14.559+0000 7fce0259c640 1 -- 172.21.15.1:0/210607512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce04101b50 msgr2=0x7fce04102570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:40:14.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:14.559+0000 7fce0259c640 1 --2- 172.21.15.1:0/210607512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce04101b50 0x7fce04102570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:40:14.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:14.559+0000 7fce0259c640 1 -- 172.21.15.1:0/210607512 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdf8005040 con 0x7fce041011b0 2024-06-18T02:40:14.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:14.559+0000 7fce0259c640 1 --2- 172.21.15.1:0/210607512 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce041011b0 0x7fce04102030 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fcdf400b790 tx=0x7fcdf40091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:40:14.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:14.560+0000 7fcdf2ffd640 1 -- 172.21.15.1:0/210607512 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcdf400c700 con 0x7fce041011b0 2024-06-18T02:40:14.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:14.560+0000 7fcdf2ffd640 1 -- 172.21.15.1:0/210607512 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcdf400cd30 con 0x7fce041011b0 2024-06-18T02:40:14.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:14.560+0000 7fcdf2ffd640 1 -- 172.21.15.1:0/210607512 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcdf4013370 con 0x7fce041011b0 2024-06-18T02:40:14.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:14.560+0000 7fce08807640 1 -- 172.21.15.1:0/210607512 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce041193c0 con 0x7fce041011b0 2024-06-18T02:40:14.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:14.560+0000 7fce08807640 1 -- 172.21.15.1:0/210607512 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce041197b0 con 0x7fce041011b0 2024-06-18T02:40:14.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:14.561+0000 7fce08807640 1 -- 172.21.15.1:0/210607512 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce040fc920 con 0x7fce041011b0 2024-06-18T02:40:14.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:14.561+0000 7fcdf2ffd640 1 -- 172.21.15.1:0/210607512 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7fcdf4012070 con 0x7fce041011b0 2024-06-18T02:40:14.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:14.561+0000 7fcdf2ffd640 1 --2- 172.21.15.1:0/210607512 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fcdec065430 0x7fcdec0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:40:14.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:14.562+0000 7fcdf2ffd640 1 -- 172.21.15.1:0/210607512 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fcdf4081cc0 con 0x7fce041011b0 2024-06-18T02:40:14.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:14.562+0000 7fce01d9b640 1 --2- 172.21.15.1:0/210607512 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fcdec065430 0x7fcdec0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:40:14.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:14.562+0000 7fce01d9b640 1 --2- 172.21.15.1:0/210607512 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fcdec065430 0x7fcdec0678e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fcdf8009f80 tx=0x7fcdf8035040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:40:14.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:14.566+0000 7fcdf2ffd640 1 -- 172.21.15.1:0/210607512 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fcdf40501f0 con 0x7fce041011b0 2024-06-18T02:40:14.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:14.774+0000 7fce08807640 1 -- 172.21.15.1:0/210607512 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fce0404b510 con 0x7fce041011b0 2024-06-18T02:40:14.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:14.774+0000 7fcdf2ffd640 1 -- 172.21.15.1:0/210607512 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fcdf4053810 con 0x7fce041011b0 2024-06-18T02:40:14.774 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:40:14.774 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-18T02:40:14.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:14.776+0000 7fce08807640 1 -- 172.21.15.1:0/210607512 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fcdec065430 msgr2=0x7fcdec0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:40:14.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:14.776+0000 7fce08807640 1 --2- 172.21.15.1:0/210607512 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fcdec065430 0x7fcdec0678e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fcdf8009f80 tx=0x7fcdf8035040 comp rx=0 tx=0).stop 2024-06-18T02:40:14.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:14.776+0000 7fce08807640 1 -- 172.21.15.1:0/210607512 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce041011b0 msgr2=0x7fce04102030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:40:14.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:14.776+0000 7fce08807640 1 --2- 172.21.15.1:0/210607512 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce041011b0 0x7fce04102030 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fcdf400b790 tx=0x7fcdf40091d0 comp rx=0 tx=0).stop 2024-06-18T02:40:14.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:14.776+0000 7fce08807640 1 -- 172.21.15.1:0/210607512 shutdown_connections 2024-06-18T02:40:14.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:14.776+0000 7fce08807640 1 --2- 172.21.15.1:0/210607512 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7fcdec065430 0x7fcdec0678e0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:40:14.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:14.776+0000 7fce08807640 1 --2- 172.21.15.1:0/210607512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce04101b50 0x7fce04102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:40:14.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:14.776+0000 7fce08807640 1 --2- 172.21.15.1:0/210607512 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce041011b0 0x7fce04102030 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:40:14.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:14.776+0000 7fce08807640 1 -- 172.21.15.1:0/210607512 >> 172.21.15.1:0/210607512 conn(0x7fce040f6d60 msgr2=0x7fce040657d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:40:14.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:14.776+0000 7fce08807640 1 -- 172.21.15.1:0/210607512 shutdown_connections 2024-06-18T02:40:14.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:14.776+0000 7fce08807640 1 -- 172.21.15.1:0/210607512 wait complete. 2024-06-18T02:40:15.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:40:15 smithi028 ceph-mon[35823]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:40:15.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:40:15 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/210607512' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:40:16.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:40:15 smithi001 ceph-mon[29099]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:40:16.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:40:15 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/210607512' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:40:17.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:40:17 smithi028 ceph-mon[35823]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:40:18.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:40:17 smithi001 ceph-mon[29099]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:40:18.377 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T02:40:18.653 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:40:19.763 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:40:19 smithi001 ceph-mon[29099]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:40:19.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:40:19 smithi028 ceph-mon[35823]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:40:20.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:20.257+0000 7f2375ded640 1 -- 172.21.15.1:0/489810403 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23701022c0 msgr2=0x7f2370102690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:40:20.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:20.257+0000 7f2375ded640 1 --2- 172.21.15.1:0/489810403 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23701022c0 0x7f2370102690 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f23600048d0 tx=0x7f2360009da0 comp rx=0 tx=0).stop 2024-06-18T02:40:20.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:20.257+0000 7f2375ded640 1 -- 172.21.15.1:0/489810403 shutdown_connections 2024-06-18T02:40:20.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:20.257+0000 7f2375ded640 1 --2- 172.21.15.1:0/489810403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2370102c60 0x7f23700fc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:40:20.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:20.257+0000 7f2375ded640 1 --2- 172.21.15.1:0/489810403 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23701022c0 0x7f2370102690 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:40:20.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:20.257+0000 7f2375ded640 1 -- 172.21.15.1:0/489810403 >> 172.21.15.1:0/489810403 conn(0x7f23700f7e50 msgr2=0x7f23700fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:40:20.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:20.257+0000 7f2375ded640 1 -- 172.21.15.1:0/489810403 shutdown_connections 2024-06-18T02:40:20.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:20.257+0000 7f2375ded640 1 -- 172.21.15.1:0/489810403 wait complete. 2024-06-18T02:40:20.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:20.258+0000 7f2375ded640 1 Processor -- start 2024-06-18T02:40:20.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:20.258+0000 7f2375ded640 1 -- start start 2024-06-18T02:40:20.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:20.258+0000 7f2375ded640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23701022c0 0x7f2370113330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:40:20.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:20.258+0000 7f2375ded640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2370102c60 0x7f2370113870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:40:20.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:20.258+0000 7f2375ded640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2370113e70 con 0x7f23701022c0 2024-06-18T02:40:20.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:20.258+0000 7f2375ded640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f237010d2b0 con 0x7f2370102c60 2024-06-18T02:40:20.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:20.258+0000 7f236f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23701022c0 0x7f2370113330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:40:20.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:20.258+0000 7f236f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23701022c0 0x7f2370113330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47982/0 (socket says 172.21.15.1:47982) 2024-06-18T02:40:20.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:20.258+0000 7f236f7fe640 1 -- 172.21.15.1:0/2231695589 learned_addr learned my addr 172.21.15.1:0/2231695589 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:40:20.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:20.258+0000 7f236effd640 1 --2- 172.21.15.1:0/2231695589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2370102c60 0x7f2370113870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:40:20.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:20.259+0000 7f236f7fe640 1 -- 172.21.15.1:0/2231695589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2370102c60 msgr2=0x7f2370113870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:40:20.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:20.259+0000 7f236f7fe640 1 --2- 172.21.15.1:0/2231695589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2370102c60 0x7f2370113870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:40:20.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:20.259+0000 7f236f7fe640 1 -- 172.21.15.1:0/2231695589 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2360005040 con 0x7f23701022c0 2024-06-18T02:40:20.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:20.259+0000 7f236f7fe640 1 --2- 172.21.15.1:0/2231695589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23701022c0 0x7f2370113330 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f23600043d0 tx=0x7f2360009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:40:20.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:20.259+0000 7f2357fff640 1 -- 172.21.15.1:0/2231695589 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2360038070 con 0x7f23701022c0 2024-06-18T02:40:20.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:20.259+0000 7f2357fff640 1 -- 172.21.15.1:0/2231695589 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2360032970 con 0x7f23701022c0 2024-06-18T02:40:20.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:20.259+0000 7f2375ded640 1 -- 172.21.15.1:0/2231695589 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f237010d530 con 0x7f23701022c0 2024-06-18T02:40:20.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:20.259+0000 7f2357fff640 1 -- 172.21.15.1:0/2231695589 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f236003da60 con 0x7f23701022c0 2024-06-18T02:40:20.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:20.259+0000 7f2375ded640 1 -- 172.21.15.1:0/2231695589 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f237010da70 con 0x7f23701022c0 2024-06-18T02:40:20.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:20.260+0000 7f2375ded640 1 -- 172.21.15.1:0/2231695589 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23700fda30 con 0x7f23701022c0 2024-06-18T02:40:20.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:20.260+0000 7f2357fff640 1 -- 172.21.15.1:0/2231695589 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7f2360030020 con 0x7f23701022c0 2024-06-18T02:40:20.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:20.261+0000 7f2357fff640 1 --2- 172.21.15.1:0/2231695589 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f2358065430 0x7f23580678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:40:20.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:20.261+0000 7f2357fff640 1 -- 172.21.15.1:0/2231695589 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f23600a2520 con 0x7f23701022c0 2024-06-18T02:40:20.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:20.261+0000 7f236effd640 1 --2- 172.21.15.1:0/2231695589 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f2358065430 0x7f23580678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:40:20.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:20.261+0000 7f236effd640 1 --2- 172.21.15.1:0/2231695589 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f2358065430 0x7f23580678e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f237010ebf0 tx=0x7f236400b040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:40:20.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:20.265+0000 7f2357fff640 1 -- 172.21.15.1:0/2231695589 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f23600709d0 con 0x7f23701022c0 2024-06-18T02:40:20.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:20.480+0000 7f2375ded640 1 -- 172.21.15.1:0/2231695589 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f2370109400 con 0x7f23701022c0 2024-06-18T02:40:20.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:20.480+0000 7f2357fff640 1 -- 172.21.15.1:0/2231695589 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f2360073ff0 con 0x7f23701022c0 2024-06-18T02:40:20.480 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:40:20.480 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-18T02:40:20.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:20.482+0000 7f2375ded640 1 -- 172.21.15.1:0/2231695589 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f2358065430 msgr2=0x7f23580678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:40:20.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:20.482+0000 7f2375ded640 1 --2- 172.21.15.1:0/2231695589 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f2358065430 0x7f23580678e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f237010ebf0 tx=0x7f236400b040 comp rx=0 tx=0).stop 2024-06-18T02:40:20.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:20.482+0000 7f2375ded640 1 -- 172.21.15.1:0/2231695589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23701022c0 msgr2=0x7f2370113330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:40:20.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:20.482+0000 7f2375ded640 1 --2- 172.21.15.1:0/2231695589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23701022c0 0x7f2370113330 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f23600043d0 tx=0x7f2360009da0 comp rx=0 tx=0).stop 2024-06-18T02:40:20.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:20.482+0000 7f2375ded640 1 -- 172.21.15.1:0/2231695589 shutdown_connections 2024-06-18T02:40:20.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:20.482+0000 7f2375ded640 1 --2- 172.21.15.1:0/2231695589 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f2358065430 0x7f23580678e0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:40:20.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:20.482+0000 7f2375ded640 1 --2- 172.21.15.1:0/2231695589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2370102c60 0x7f2370113870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:40:20.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:20.482+0000 7f2375ded640 1 --2- 172.21.15.1:0/2231695589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23701022c0 0x7f2370113330 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:40:20.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:20.482+0000 7f2375ded640 1 -- 172.21.15.1:0/2231695589 >> 172.21.15.1:0/2231695589 conn(0x7f23700f7e50 msgr2=0x7f23700f8de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:40:20.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:20.482+0000 7f2375ded640 1 -- 172.21.15.1:0/2231695589 shutdown_connections 2024-06-18T02:40:20.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:20.482+0000 7f2375ded640 1 -- 172.21.15.1:0/2231695589 wait complete. 2024-06-18T02:40:20.933 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:40:20 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/2231695589' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:40:20.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:40:20 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/2231695589' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:40:21.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:40:21 smithi001 ceph-mon[29099]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:40:21.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:40:21 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:40:21.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:40:21 smithi028 ceph-mon[35823]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:40:21.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:40:21 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:40:23.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:40:23 smithi001 ceph-mon[29099]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:40:23.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:40:23 smithi028 ceph-mon[35823]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:40:24.262 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T02:40:24.540 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:40:25.945 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:40:25 smithi001 ceph-mon[29099]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:40:25.945 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:40:25 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:40:25.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:40:25 smithi028 ceph-mon[35823]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:40:25.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:40:25 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:40:26.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:26.260+0000 7f5a6c4ad640 1 -- 172.21.15.1:0/3832431360 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a641022f0 msgr2=0x7f5a641026c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:40:26.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:26.260+0000 7f5a6c4ad640 1 --2- 172.21.15.1:0/3832431360 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a641022f0 0x7f5a641026c0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f5a600048d0 tx=0x7f5a60009da0 comp rx=0 tx=0).stop 2024-06-18T02:40:26.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:26.260+0000 7f5a6c4ad640 1 -- 172.21.15.1:0/3832431360 shutdown_connections 2024-06-18T02:40:26.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:26.260+0000 7f5a6c4ad640 1 --2- 172.21.15.1:0/3832431360 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a64102c90 0x7f5a640fc320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:40:26.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:26.260+0000 7f5a6c4ad640 1 --2- 172.21.15.1:0/3832431360 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a641022f0 0x7f5a641026c0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:40:26.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:26.260+0000 7f5a6c4ad640 1 -- 172.21.15.1:0/3832431360 >> 172.21.15.1:0/3832431360 conn(0x7f5a640f7f00 msgr2=0x7f5a640fa310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:40:26.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:26.260+0000 7f5a6c4ad640 1 -- 172.21.15.1:0/3832431360 shutdown_connections 2024-06-18T02:40:26.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:26.260+0000 7f5a6c4ad640 1 -- 172.21.15.1:0/3832431360 wait complete. 2024-06-18T02:40:26.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:26.260+0000 7f5a6c4ad640 1 Processor -- start 2024-06-18T02:40:26.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:26.260+0000 7f5a6c4ad640 1 -- start start 2024-06-18T02:40:26.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:26.260+0000 7f5a6c4ad640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a641022f0 0x7f5a6406f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:40:26.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:26.260+0000 7f5a6c4ad640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a64102c90 0x7f5a6406fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:40:26.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:26.260+0000 7f5a6c4ad640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a640701f0 con 0x7f5a641022f0 2024-06-18T02:40:26.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:26.261+0000 7f5a6c4ad640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a64070360 con 0x7f5a64102c90 2024-06-18T02:40:26.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:26.261+0000 7f5a6a249640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a641022f0 0x7f5a6406f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:40:26.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:26.261+0000 7f5a6a249640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a641022f0 0x7f5a6406f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43444/0 (socket says 172.21.15.1:43444) 2024-06-18T02:40:26.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:26.261+0000 7f5a6a249640 1 -- 172.21.15.1:0/2783854805 learned_addr learned my addr 172.21.15.1:0/2783854805 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:40:26.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:26.261+0000 7f5a69a48640 1 --2- 172.21.15.1:0/2783854805 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a64102c90 0x7f5a6406fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:40:26.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:26.261+0000 7f5a6a249640 1 -- 172.21.15.1:0/2783854805 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a64102c90 msgr2=0x7f5a6406fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:40:26.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:26.261+0000 7f5a6a249640 1 --2- 172.21.15.1:0/2783854805 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a64102c90 0x7f5a6406fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:40:26.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:26.261+0000 7f5a6a249640 1 -- 172.21.15.1:0/2783854805 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a60005040 con 0x7f5a641022f0 2024-06-18T02:40:26.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:26.261+0000 7f5a69a48640 1 --2- 172.21.15.1:0/2783854805 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a64102c90 0x7f5a6406fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-18T02:40:26.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:26.261+0000 7f5a6a249640 1 --2- 172.21.15.1:0/2783854805 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a641022f0 0x7f5a6406f770 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f5a600043d0 tx=0x7f5a60009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:40:26.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:26.261+0000 7f5a5affd640 1 -- 172.21.15.1:0/2783854805 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5a60038070 con 0x7f5a641022f0 2024-06-18T02:40:26.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:26.261+0000 7f5a6c4ad640 1 -- 172.21.15.1:0/2783854805 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a64070590 con 0x7f5a641022f0 2024-06-18T02:40:26.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:26.261+0000 7f5a5affd640 1 -- 172.21.15.1:0/2783854805 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5a60032970 con 0x7f5a641022f0 2024-06-18T02:40:26.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:26.261+0000 7f5a5affd640 1 -- 172.21.15.1:0/2783854805 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5a6003da60 con 0x7f5a641022f0 2024-06-18T02:40:26.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:26.261+0000 7f5a6c4ad640 1 -- 172.21.15.1:0/2783854805 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a64119660 con 0x7f5a641022f0 2024-06-18T02:40:26.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:26.262+0000 7f5a6c4ad640 1 -- 172.21.15.1:0/2783854805 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a640fda60 con 0x7f5a641022f0 2024-06-18T02:40:26.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:26.263+0000 7f5a5affd640 1 -- 172.21.15.1:0/2783854805 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7f5a60030020 con 0x7f5a641022f0 2024-06-18T02:40:26.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:26.263+0000 7f5a5affd640 1 --2- 172.21.15.1:0/2783854805 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f5a50065430 0x7f5a500678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:40:26.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:26.263+0000 7f5a5affd640 1 -- 172.21.15.1:0/2783854805 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f5a600a25c0 con 0x7f5a641022f0 2024-06-18T02:40:26.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:26.263+0000 7f5a69a48640 1 --2- 172.21.15.1:0/2783854805 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f5a50065430 0x7f5a500678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:40:26.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:26.264+0000 7f5a69a48640 1 --2- 172.21.15.1:0/2783854805 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f5a50065430 0x7f5a500678e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5a54007fd0 tx=0x7f5a54009e90 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:40:26.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:26.269+0000 7f5a5affd640 1 -- 172.21.15.1:0/2783854805 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f5a60070a70 con 0x7f5a641022f0 2024-06-18T02:40:26.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:26.482+0000 7f5a6c4ad640 1 -- 172.21.15.1:0/2783854805 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f5a640fbb00 con 0x7f5a641022f0 2024-06-18T02:40:26.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:26.483+0000 7f5a5affd640 1 -- 172.21.15.1:0/2783854805 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f5a60074090 con 0x7f5a641022f0 2024-06-18T02:40:26.483 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:40:26.483 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-18T02:40:26.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:26.485+0000 7f5a6c4ad640 1 -- 172.21.15.1:0/2783854805 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f5a50065430 msgr2=0x7f5a500678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:40:26.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:26.485+0000 7f5a6c4ad640 1 --2- 172.21.15.1:0/2783854805 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f5a50065430 0x7f5a500678e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5a54007fd0 tx=0x7f5a54009e90 comp rx=0 tx=0).stop 2024-06-18T02:40:26.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:26.485+0000 7f5a6c4ad640 1 -- 172.21.15.1:0/2783854805 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a641022f0 msgr2=0x7f5a6406f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:40:26.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:26.485+0000 7f5a6c4ad640 1 --2- 172.21.15.1:0/2783854805 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a641022f0 0x7f5a6406f770 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f5a600043d0 tx=0x7f5a60009da0 comp rx=0 tx=0).stop 2024-06-18T02:40:26.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:26.485+0000 7f5a6c4ad640 1 -- 172.21.15.1:0/2783854805 shutdown_connections 2024-06-18T02:40:26.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:26.485+0000 7f5a6c4ad640 1 --2- 172.21.15.1:0/2783854805 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f5a50065430 0x7f5a500678e0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:40:26.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:26.485+0000 7f5a6c4ad640 1 --2- 172.21.15.1:0/2783854805 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a64102c90 0x7f5a6406fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:40:26.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:26.485+0000 7f5a6c4ad640 1 --2- 172.21.15.1:0/2783854805 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a641022f0 0x7f5a6406f770 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:40:26.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:26.485+0000 7f5a6c4ad640 1 -- 172.21.15.1:0/2783854805 >> 172.21.15.1:0/2783854805 conn(0x7f5a640f7f00 msgr2=0x7f5a640f8e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:40:26.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:26.485+0000 7f5a6c4ad640 1 -- 172.21.15.1:0/2783854805 shutdown_connections 2024-06-18T02:40:26.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:26.485+0000 7f5a6c4ad640 1 -- 172.21.15.1:0/2783854805 wait complete. 2024-06-18T02:40:26.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:40:26 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:40:26.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:40:26 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T02:40:26.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:40:26 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:40:26.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:40:26 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/2783854805' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:40:27.077 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:40:26 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:40:27.077 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:40:26 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T02:40:27.077 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:40:26 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' 2024-06-18T02:40:27.077 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:40:26 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/2783854805' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:40:27.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:40:27 smithi028 ceph-mon[35823]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:40:28.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:40:27 smithi001 ceph-mon[29099]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:40:29.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:40:29 smithi028 ceph-mon[35823]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:40:30.103 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T02:40:30.127 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:40:29 smithi001 ceph-mon[29099]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:40:30.380 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:40:31.803 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:40:31 smithi001 ceph-mon[29099]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:40:31.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:40:31 smithi028 ceph-mon[35823]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:40:31.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:31.993+0000 7f78f7e50640 1 -- 172.21.15.1:0/889085561 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78f01022a0 msgr2=0x7f78f0102670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:40:31.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:31.993+0000 7f78f7e50640 1 --2- 172.21.15.1:0/889085561 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78f01022a0 0x7f78f0102670 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f78ec0048d0 tx=0x7f78ec009da0 comp rx=0 tx=0).stop 2024-06-18T02:40:31.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:31.994+0000 7f78f7e50640 1 -- 172.21.15.1:0/889085561 shutdown_connections 2024-06-18T02:40:31.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:31.994+0000 7f78f7e50640 1 --2- 172.21.15.1:0/889085561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78f0102c40 0x7f78f00fc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:40:31.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:31.994+0000 7f78f7e50640 1 --2- 172.21.15.1:0/889085561 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78f01022a0 0x7f78f0102670 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:40:31.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:31.994+0000 7f78f7e50640 1 -- 172.21.15.1:0/889085561 >> 172.21.15.1:0/889085561 conn(0x7f78f00f7e70 msgr2=0x7f78f00fa280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:40:31.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:31.994+0000 7f78f7e50640 1 -- 172.21.15.1:0/889085561 shutdown_connections 2024-06-18T02:40:31.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:31.994+0000 7f78f7e50640 1 -- 172.21.15.1:0/889085561 wait complete. 2024-06-18T02:40:31.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:31.994+0000 7f78f7e50640 1 Processor -- start 2024-06-18T02:40:31.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:31.994+0000 7f78f7e50640 1 -- start start 2024-06-18T02:40:31.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:31.995+0000 7f78f7e50640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78f01022a0 0x7f78f0112fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:40:31.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:31.995+0000 7f78f7e50640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78f0102c40 0x7f78f0113520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:40:31.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:31.995+0000 7f78f7e50640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f0113b20 con 0x7f78f01022a0 2024-06-18T02:40:31.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:31.995+0000 7f78f7e50640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f010cf60 con 0x7f78f0102c40 2024-06-18T02:40:31.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:31.995+0000 7f78f5bec640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78f01022a0 0x7f78f0112fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:40:31.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:31.995+0000 7f78f5bec640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78f01022a0 0x7f78f0112fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43468/0 (socket says 172.21.15.1:43468) 2024-06-18T02:40:31.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:31.995+0000 7f78f5bec640 1 -- 172.21.15.1:0/3246009383 learned_addr learned my addr 172.21.15.1:0/3246009383 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:40:31.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:31.995+0000 7f78f53eb640 1 --2- 172.21.15.1:0/3246009383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78f0102c40 0x7f78f0113520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:40:31.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:31.995+0000 7f78f5bec640 1 -- 172.21.15.1:0/3246009383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78f0102c40 msgr2=0x7f78f0113520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:40:31.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:31.995+0000 7f78f5bec640 1 --2- 172.21.15.1:0/3246009383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78f0102c40 0x7f78f0113520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:40:31.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:31.995+0000 7f78f5bec640 1 -- 172.21.15.1:0/3246009383 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78ec005040 con 0x7f78f01022a0 2024-06-18T02:40:31.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:31.995+0000 7f78f5bec640 1 --2- 172.21.15.1:0/3246009383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78f01022a0 0x7f78f0112fe0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f78ec0043d0 tx=0x7f78ec009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:40:31.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:31.996+0000 7f78e67fc640 1 -- 172.21.15.1:0/3246009383 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f78ec038070 con 0x7f78f01022a0 2024-06-18T02:40:31.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:31.996+0000 7f78e67fc640 1 -- 172.21.15.1:0/3246009383 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f78ec032970 con 0x7f78f01022a0 2024-06-18T02:40:31.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:31.996+0000 7f78f7e50640 1 -- 172.21.15.1:0/3246009383 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78f010d1e0 con 0x7f78f01022a0 2024-06-18T02:40:31.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:31.996+0000 7f78e67fc640 1 -- 172.21.15.1:0/3246009383 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f78ec03da60 con 0x7f78f01022a0 2024-06-18T02:40:31.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:31.996+0000 7f78f7e50640 1 -- 172.21.15.1:0/3246009383 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78f010d6a0 con 0x7f78f01022a0 2024-06-18T02:40:31.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:31.997+0000 7f78f7e50640 1 -- 172.21.15.1:0/3246009383 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78f00fda10 con 0x7f78f01022a0 2024-06-18T02:40:31.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:31.997+0000 7f78e67fc640 1 -- 172.21.15.1:0/3246009383 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7f78ec030020 con 0x7f78f01022a0 2024-06-18T02:40:31.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:31.998+0000 7f78e67fc640 1 --2- 172.21.15.1:0/3246009383 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f78dc065430 0x7f78dc0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:40:31.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:31.998+0000 7f78e67fc640 1 -- 172.21.15.1:0/3246009383 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f78ec0a25b0 con 0x7f78f01022a0 2024-06-18T02:40:31.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:31.998+0000 7f78f53eb640 1 --2- 172.21.15.1:0/3246009383 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f78dc065430 0x7f78dc0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:40:31.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:31.998+0000 7f78f53eb640 1 --2- 172.21.15.1:0/3246009383 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f78dc065430 0x7f78dc0678e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f78f010e8a0 tx=0x7f78e000b040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:40:32.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:32.002+0000 7f78e67fc640 1 -- 172.21.15.1:0/3246009383 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f78ec070a60 con 0x7f78f01022a0 2024-06-18T02:40:32.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:32.209+0000 7f78f7e50640 1 -- 172.21.15.1:0/3246009383 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f78f01093e0 con 0x7f78f01022a0 2024-06-18T02:40:32.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:32.209+0000 7f78e67fc640 1 -- 172.21.15.1:0/3246009383 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f78ec074080 con 0x7f78f01022a0 2024-06-18T02:40:32.209 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:40:32.209 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-18T02:40:32.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:32.211+0000 7f78f7e50640 1 -- 172.21.15.1:0/3246009383 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f78dc065430 msgr2=0x7f78dc0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:40:32.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:32.211+0000 7f78f7e50640 1 --2- 172.21.15.1:0/3246009383 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f78dc065430 0x7f78dc0678e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f78f010e8a0 tx=0x7f78e000b040 comp rx=0 tx=0).stop 2024-06-18T02:40:32.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:32.211+0000 7f78f7e50640 1 -- 172.21.15.1:0/3246009383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78f01022a0 msgr2=0x7f78f0112fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:40:32.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:32.211+0000 7f78f7e50640 1 --2- 172.21.15.1:0/3246009383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78f01022a0 0x7f78f0112fe0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f78ec0043d0 tx=0x7f78ec009da0 comp rx=0 tx=0).stop 2024-06-18T02:40:32.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:32.211+0000 7f78f7e50640 1 -- 172.21.15.1:0/3246009383 shutdown_connections 2024-06-18T02:40:32.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:32.211+0000 7f78f7e50640 1 --2- 172.21.15.1:0/3246009383 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f78dc065430 0x7f78dc0678e0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:40:32.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:32.211+0000 7f78f7e50640 1 --2- 172.21.15.1:0/3246009383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78f0102c40 0x7f78f0113520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:40:32.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:32.211+0000 7f78f7e50640 1 --2- 172.21.15.1:0/3246009383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78f01022a0 0x7f78f0112fe0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:40:32.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:32.211+0000 7f78f7e50640 1 -- 172.21.15.1:0/3246009383 >> 172.21.15.1:0/3246009383 conn(0x7f78f00f7e70 msgr2=0x7f78f00f8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:40:32.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:32.211+0000 7f78f7e50640 1 -- 172.21.15.1:0/3246009383 shutdown_connections 2024-06-18T02:40:32.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:32.211+0000 7f78f7e50640 1 -- 172.21.15.1:0/3246009383 wait complete. 2024-06-18T02:40:32.838 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:40:32 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/3246009383' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:40:32.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:40:32 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/3246009383' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:40:33.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:40:33 smithi001 ceph-mon[29099]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:40:33.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:40:33 smithi028 ceph-mon[35823]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:40:35.841 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T02:40:35.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:40:35 smithi028 ceph-mon[35823]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:40:36.117 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:40:36.127 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:40:35 smithi001 ceph-mon[29099]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:40:36.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:40:36 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:40:36.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:40:36 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:40:37.830 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:40:37 smithi001 ceph-mon[29099]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:40:37.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:37.829+0000 7ff12f091640 1 -- 172.21.15.1:0/891125193 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1281011a0 msgr2=0x7ff128101570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:40:37.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:37.829+0000 7ff12f091640 1 --2- 172.21.15.1:0/891125193 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1281011a0 0x7ff128101570 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7ff1180048d0 tx=0x7ff118009da0 comp rx=0 tx=0).stop 2024-06-18T02:40:37.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:37.830+0000 7ff12f091640 1 -- 172.21.15.1:0/891125193 shutdown_connections 2024-06-18T02:40:37.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:37.830+0000 7ff12f091640 1 --2- 172.21.15.1:0/891125193 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff128101b40 0x7ff1280fb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:40:37.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:37.830+0000 7ff12f091640 1 --2- 172.21.15.1:0/891125193 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1281011a0 0x7ff128101570 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:40:37.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:37.830+0000 7ff12f091640 1 -- 172.21.15.1:0/891125193 >> 172.21.15.1:0/891125193 conn(0x7ff1280f6d90 msgr2=0x7ff1280f91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:40:37.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:37.830+0000 7ff12f091640 1 -- 172.21.15.1:0/891125193 shutdown_connections 2024-06-18T02:40:37.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:37.830+0000 7ff12f091640 1 -- 172.21.15.1:0/891125193 wait complete. 2024-06-18T02:40:37.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:37.830+0000 7ff12f091640 1 Processor -- start 2024-06-18T02:40:37.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:37.830+0000 7ff12f091640 1 -- start start 2024-06-18T02:40:37.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:37.831+0000 7ff12f091640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1281011a0 0x7ff128115210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:40:37.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:37.831+0000 7ff12f091640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff128101b40 0x7ff128115750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:40:37.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:37.831+0000 7ff12f091640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff128115d50 con 0x7ff1281011a0 2024-06-18T02:40:37.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:37.831+0000 7ff12f091640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff12810f190 con 0x7ff128101b40 2024-06-18T02:40:37.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:37.831+0000 7ff127fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff128101b40 0x7ff128115750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:40:37.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:37.831+0000 7ff127fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff128101b40 0x7ff128115750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.1:51526/0 (socket says 172.21.15.1:51526) 2024-06-18T02:40:37.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:37.831+0000 7ff127fff640 1 -- 172.21.15.1:0/2279909361 learned_addr learned my addr 172.21.15.1:0/2279909361 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:40:37.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:37.831+0000 7ff12ce2d640 1 --2- 172.21.15.1:0/2279909361 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1281011a0 0x7ff128115210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:40:37.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:37.831+0000 7ff12ce2d640 1 -- 172.21.15.1:0/2279909361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff128101b40 msgr2=0x7ff128115750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:40:37.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:37.831+0000 7ff12ce2d640 1 --2- 172.21.15.1:0/2279909361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff128101b40 0x7ff128115750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:40:37.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:37.831+0000 7ff12ce2d640 1 -- 172.21.15.1:0/2279909361 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff118005040 con 0x7ff1281011a0 2024-06-18T02:40:37.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:37.831+0000 7ff12ce2d640 1 --2- 172.21.15.1:0/2279909361 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1281011a0 0x7ff128115210 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7ff118000c00 tx=0x7ff118009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:40:37.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:37.832+0000 7ff1257fa640 1 -- 172.21.15.1:0/2279909361 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff118038070 con 0x7ff1281011a0 2024-06-18T02:40:37.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:37.832+0000 7ff1257fa640 1 -- 172.21.15.1:0/2279909361 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff11802abd0 con 0x7ff1281011a0 2024-06-18T02:40:37.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:37.832+0000 7ff12f091640 1 -- 172.21.15.1:0/2279909361 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff12810f410 con 0x7ff1281011a0 2024-06-18T02:40:37.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:37.832+0000 7ff1257fa640 1 -- 172.21.15.1:0/2279909361 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff118033640 con 0x7ff1281011a0 2024-06-18T02:40:37.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:37.832+0000 7ff12f091640 1 -- 172.21.15.1:0/2279909361 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff12810f8c0 con 0x7ff1281011a0 2024-06-18T02:40:37.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:37.833+0000 7ff12f091640 1 -- 172.21.15.1:0/2279909361 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1280fc910 con 0x7ff1281011a0 2024-06-18T02:40:37.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:37.833+0000 7ff1257fa640 1 -- 172.21.15.1:0/2279909361 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7ff118030020 con 0x7ff1281011a0 2024-06-18T02:40:37.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:37.833+0000 7ff1257fa640 1 --2- 172.21.15.1:0/2279909361 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7ff110065430 0x7ff1100678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:40:37.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:37.834+0000 7ff1257fa640 1 -- 172.21.15.1:0/2279909361 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7ff1180a1ea0 con 0x7ff1281011a0 2024-06-18T02:40:37.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:37.834+0000 7ff127fff640 1 --2- 172.21.15.1:0/2279909361 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7ff110065430 0x7ff1100678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:40:37.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:37.834+0000 7ff127fff640 1 --2- 172.21.15.1:0/2279909361 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7ff110065430 0x7ff1100678e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff128110ad0 tx=0x7ff11c00b040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:40:37.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:37.838+0000 7ff1257fa640 1 -- 172.21.15.1:0/2279909361 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7ff118070440 con 0x7ff1281011a0 2024-06-18T02:40:37.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:40:37 smithi028 ceph-mon[35823]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:40:38.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:38.045+0000 7ff12f091640 1 -- 172.21.15.1:0/2279909361 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff12804b510 con 0x7ff1281011a0 2024-06-18T02:40:38.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:38.046+0000 7ff1257fa640 1 -- 172.21.15.1:0/2279909361 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7ff118073a60 con 0x7ff1281011a0 2024-06-18T02:40:38.046 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:40:38.046 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-18T02:40:38.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:38.047+0000 7ff12f091640 1 -- 172.21.15.1:0/2279909361 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7ff110065430 msgr2=0x7ff1100678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:40:38.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:38.047+0000 7ff12f091640 1 --2- 172.21.15.1:0/2279909361 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7ff110065430 0x7ff1100678e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff128110ad0 tx=0x7ff11c00b040 comp rx=0 tx=0).stop 2024-06-18T02:40:38.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:38.047+0000 7ff12f091640 1 -- 172.21.15.1:0/2279909361 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1281011a0 msgr2=0x7ff128115210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:40:38.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:38.047+0000 7ff12f091640 1 --2- 172.21.15.1:0/2279909361 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1281011a0 0x7ff128115210 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7ff118000c00 tx=0x7ff118009da0 comp rx=0 tx=0).stop 2024-06-18T02:40:38.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:38.048+0000 7ff12f091640 1 -- 172.21.15.1:0/2279909361 shutdown_connections 2024-06-18T02:40:38.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:38.048+0000 7ff12f091640 1 --2- 172.21.15.1:0/2279909361 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7ff110065430 0x7ff1100678e0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:40:38.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:38.048+0000 7ff12f091640 1 --2- 172.21.15.1:0/2279909361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff128101b40 0x7ff128115750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:40:38.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:38.048+0000 7ff12f091640 1 --2- 172.21.15.1:0/2279909361 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1281011a0 0x7ff128115210 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:40:38.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:38.048+0000 7ff12f091640 1 -- 172.21.15.1:0/2279909361 >> 172.21.15.1:0/2279909361 conn(0x7ff1280f6d90 msgr2=0x7ff1280f8400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:40:38.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:38.048+0000 7ff12f091640 1 -- 172.21.15.1:0/2279909361 shutdown_connections 2024-06-18T02:40:38.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:38.048+0000 7ff12f091640 1 -- 172.21.15.1:0/2279909361 wait complete. 2024-06-18T02:40:38.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:40:38 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/2279909361' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:40:39.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:40:38 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/2279909361' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:40:39.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:40:39 smithi028 ceph-mon[35823]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:40:40.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:40:39 smithi001 ceph-mon[29099]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:40:41.764 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T02:40:41.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:40:41 smithi028 ceph-mon[35823]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:40:42.040 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:40:42.127 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:40:41 smithi001 ceph-mon[29099]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:40:43.733 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:40:43 smithi001 ceph-mon[29099]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:40:43.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:43.809+0000 7f1980335640 1 -- 172.21.15.1:0/4168982784 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19780fb090 msgr2=0x7f19780fb460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:40:43.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:43.809+0000 7f1980335640 1 --2- 172.21.15.1:0/4168982784 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19780fb090 0x7f19780fb460 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f19740048d0 tx=0x7f1974009da0 comp rx=0 tx=0).stop 2024-06-18T02:40:43.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:43.810+0000 7f1980335640 1 -- 172.21.15.1:0/4168982784 shutdown_connections 2024-06-18T02:40:43.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:43.810+0000 7f1980335640 1 --2- 172.21.15.1:0/4168982784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19780fb9a0 0x7f1978108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:40:43.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:43.810+0000 7f1980335640 1 --2- 172.21.15.1:0/4168982784 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19780fb090 0x7f19780fb460 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:40:43.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:43.810+0000 7f1980335640 1 -- 172.21.15.1:0/4168982784 >> 172.21.15.1:0/4168982784 conn(0x7f19780f6d60 msgr2=0x7f19780f9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:40:43.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:43.810+0000 7f1980335640 1 -- 172.21.15.1:0/4168982784 shutdown_connections 2024-06-18T02:40:43.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:43.810+0000 7f1980335640 1 -- 172.21.15.1:0/4168982784 wait complete. 2024-06-18T02:40:43.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:43.810+0000 7f1980335640 1 Processor -- start 2024-06-18T02:40:43.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:43.810+0000 7f1980335640 1 -- start start 2024-06-18T02:40:43.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:43.810+0000 7f1980335640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19780fb090 0x7f19781151e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:40:43.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:43.810+0000 7f1980335640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19780fb9a0 0x7f1978115720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:40:43.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:43.810+0000 7f1980335640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1978115d20 con 0x7f19780fb090 2024-06-18T02:40:43.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:43.810+0000 7f1980335640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f197810f160 con 0x7f19780fb9a0 2024-06-18T02:40:43.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:43.811+0000 7f197e0d1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19780fb090 0x7f19781151e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:40:43.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:43.811+0000 7f197e0d1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19780fb090 0x7f19781151e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47264/0 (socket says 172.21.15.1:47264) 2024-06-18T02:40:43.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:43.811+0000 7f197e0d1640 1 -- 172.21.15.1:0/834160597 learned_addr learned my addr 172.21.15.1:0/834160597 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:40:43.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:43.811+0000 7f197d8d0640 1 --2- 172.21.15.1:0/834160597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19780fb9a0 0x7f1978115720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:40:43.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:43.811+0000 7f197e0d1640 1 -- 172.21.15.1:0/834160597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19780fb9a0 msgr2=0x7f1978115720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:40:43.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:43.811+0000 7f197e0d1640 1 --2- 172.21.15.1:0/834160597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19780fb9a0 0x7f1978115720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:40:43.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:43.811+0000 7f197e0d1640 1 -- 172.21.15.1:0/834160597 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1974005040 con 0x7f19780fb090 2024-06-18T02:40:43.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:43.811+0000 7f197e0d1640 1 --2- 172.21.15.1:0/834160597 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19780fb090 0x7f19781151e0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f19740048a0 tx=0x7f1974009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:40:43.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:43.811+0000 7f196effd640 1 -- 172.21.15.1:0/834160597 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1974038070 con 0x7f19780fb090 2024-06-18T02:40:43.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:43.811+0000 7f196effd640 1 -- 172.21.15.1:0/834160597 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1974008ee0 con 0x7f19780fb090 2024-06-18T02:40:43.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:43.811+0000 7f1980335640 1 -- 172.21.15.1:0/834160597 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f197810f3e0 con 0x7f19780fb090 2024-06-18T02:40:43.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:43.811+0000 7f196effd640 1 -- 172.21.15.1:0/834160597 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f197403c770 con 0x7f19780fb090 2024-06-18T02:40:43.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:43.811+0000 7f1980335640 1 -- 172.21.15.1:0/834160597 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f197810f980 con 0x7f19780fb090 2024-06-18T02:40:43.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:43.812+0000 7f1980335640 1 -- 172.21.15.1:0/834160597 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1978066030 con 0x7f19780fb090 2024-06-18T02:40:43.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:43.813+0000 7f196effd640 1 -- 172.21.15.1:0/834160597 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7f1974030020 con 0x7f19780fb090 2024-06-18T02:40:43.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:43.813+0000 7f196effd640 1 --2- 172.21.15.1:0/834160597 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f1964065430 0x7f19640678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:40:43.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:43.813+0000 7f196effd640 1 -- 172.21.15.1:0/834160597 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f19740a20a0 con 0x7f19780fb090 2024-06-18T02:40:43.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:43.813+0000 7f197d8d0640 1 --2- 172.21.15.1:0/834160597 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f1964065430 0x7f19640678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:40:43.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:43.814+0000 7f197d8d0640 1 --2- 172.21.15.1:0/834160597 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f1964065430 0x7f19640678e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1978110aa0 tx=0x7f196800b040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:40:43.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:43.817+0000 7f196effd640 1 -- 172.21.15.1:0/834160597 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f1974070680 con 0x7f19780fb090 2024-06-18T02:40:43.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:40:43 smithi028 ceph-mon[35823]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:40:44.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:44.032+0000 7f1980335640 1 -- 172.21.15.1:0/834160597 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f197810f570 con 0x7f19780fb090 2024-06-18T02:40:44.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:44.032+0000 7f196effd640 1 -- 172.21.15.1:0/834160597 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f197810f570 con 0x7f19780fb090 2024-06-18T02:40:44.032 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:40:44.032 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-18T02:40:44.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:44.034+0000 7f1980335640 1 -- 172.21.15.1:0/834160597 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f1964065430 msgr2=0x7f19640678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:40:44.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:44.034+0000 7f1980335640 1 --2- 172.21.15.1:0/834160597 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f1964065430 0x7f19640678e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1978110aa0 tx=0x7f196800b040 comp rx=0 tx=0).stop 2024-06-18T02:40:44.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:44.034+0000 7f1980335640 1 -- 172.21.15.1:0/834160597 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19780fb090 msgr2=0x7f19781151e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:40:44.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:44.034+0000 7f1980335640 1 --2- 172.21.15.1:0/834160597 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19780fb090 0x7f19781151e0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f19740048a0 tx=0x7f1974009da0 comp rx=0 tx=0).stop 2024-06-18T02:40:44.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:44.035+0000 7f1980335640 1 -- 172.21.15.1:0/834160597 shutdown_connections 2024-06-18T02:40:44.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:44.035+0000 7f1980335640 1 --2- 172.21.15.1:0/834160597 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f1964065430 0x7f19640678e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:40:44.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:44.035+0000 7f1980335640 1 --2- 172.21.15.1:0/834160597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19780fb9a0 0x7f1978115720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:40:44.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:44.035+0000 7f1980335640 1 --2- 172.21.15.1:0/834160597 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f19780fb090 0x7f19781151e0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:40:44.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:44.035+0000 7f1980335640 1 -- 172.21.15.1:0/834160597 >> 172.21.15.1:0/834160597 conn(0x7f19780f6d60 msgr2=0x7f19780f71f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:40:44.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:44.035+0000 7f1980335640 1 -- 172.21.15.1:0/834160597 shutdown_connections 2024-06-18T02:40:44.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:44.035+0000 7f1980335640 1 -- 172.21.15.1:0/834160597 wait complete. 2024-06-18T02:40:44.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:40:44 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/834160597' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:40:44.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:40:44 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/834160597' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:40:45.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:40:45 smithi028 ceph-mon[35823]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:40:46.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:40:45 smithi001 ceph-mon[29099]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:40:47.950 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T02:40:47.979 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:40:47 smithi001 ceph-mon[29099]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:40:47.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:40:47 smithi028 ceph-mon[35823]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:40:48.229 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:40:49.956 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:40:49 smithi001 ceph-mon[29099]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:40:49.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:40:49 smithi028 ceph-mon[35823]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:40:50.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:50.034+0000 7f644c9a1640 1 -- 172.21.15.1:0/310620644 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64480fdcf0 msgr2=0x7f64481063c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:40:50.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:50.034+0000 7f644c9a1640 1 --2- 172.21.15.1:0/310620644 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64480fdcf0 0x7f64481063c0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f643c0048d0 tx=0x7f643c009da0 comp rx=0 tx=0).stop 2024-06-18T02:40:50.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:50.034+0000 7f644c9a1640 1 -- 172.21.15.1:0/310620644 shutdown_connections 2024-06-18T02:40:50.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:50.034+0000 7f644c9a1640 1 --2- 172.21.15.1:0/310620644 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64480fdcf0 0x7f64481063c0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:40:50.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:50.034+0000 7f644c9a1640 1 --2- 172.21.15.1:0/310620644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64480fd350 0x7f64480fd720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:40:50.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:50.034+0000 7f644c9a1640 1 -- 172.21.15.1:0/310620644 >> 172.21.15.1:0/310620644 conn(0x7f64480f6db0 msgr2=0x7f64480f91c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:40:50.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:50.034+0000 7f644c9a1640 1 -- 172.21.15.1:0/310620644 shutdown_connections 2024-06-18T02:40:50.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:50.034+0000 7f644c9a1640 1 -- 172.21.15.1:0/310620644 wait complete. 2024-06-18T02:40:50.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:50.034+0000 7f644c9a1640 1 Processor -- start 2024-06-18T02:40:50.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:50.034+0000 7f644c9a1640 1 -- start start 2024-06-18T02:40:50.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:50.035+0000 7f644c9a1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64480fd350 0x7f64481152c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:40:50.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:50.035+0000 7f644c9a1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64480fdcf0 0x7f6448115800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:40:50.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:50.035+0000 7f644c9a1640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6448115e00 con 0x7f64480fdcf0 2024-06-18T02:40:50.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:50.035+0000 7f644c9a1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f644810f240 con 0x7f64480fd350 2024-06-18T02:40:50.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:50.035+0000 7f6445d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64480fdcf0 0x7f6448115800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:40:50.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:50.035+0000 7f6445d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64480fdcf0 0x7f6448115800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47280/0 (socket says 172.21.15.1:47280) 2024-06-18T02:40:50.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:50.035+0000 7f6445d9b640 1 -- 172.21.15.1:0/454352144 learned_addr learned my addr 172.21.15.1:0/454352144 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:40:50.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:50.035+0000 7f644659c640 1 --2- 172.21.15.1:0/454352144 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64480fd350 0x7f64481152c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:40:50.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:50.035+0000 7f6445d9b640 1 -- 172.21.15.1:0/454352144 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64480fd350 msgr2=0x7f64481152c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:40:50.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:50.035+0000 7f6445d9b640 1 --2- 172.21.15.1:0/454352144 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64480fd350 0x7f64481152c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:40:50.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:50.035+0000 7f6445d9b640 1 -- 172.21.15.1:0/454352144 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f643c005040 con 0x7f64480fdcf0 2024-06-18T02:40:50.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:50.035+0000 7f6445d9b640 1 --2- 172.21.15.1:0/454352144 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64480fdcf0 0x7f6448115800 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f643c0048a0 tx=0x7f643c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:40:50.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:50.035+0000 7f6436ffd640 1 -- 172.21.15.1:0/454352144 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f643c038070 con 0x7f64480fdcf0 2024-06-18T02:40:50.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:50.035+0000 7f6436ffd640 1 -- 172.21.15.1:0/454352144 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f643c008ee0 con 0x7f64480fdcf0 2024-06-18T02:40:50.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:50.035+0000 7f644c9a1640 1 -- 172.21.15.1:0/454352144 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f644810f4c0 con 0x7f64480fdcf0 2024-06-18T02:40:50.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:50.035+0000 7f6436ffd640 1 -- 172.21.15.1:0/454352144 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f643c033760 con 0x7f64480fdcf0 2024-06-18T02:40:50.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:50.035+0000 7f644c9a1640 1 -- 172.21.15.1:0/454352144 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f644810f980 con 0x7f64480fdcf0 2024-06-18T02:40:50.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:50.036+0000 7f644c9a1640 1 -- 172.21.15.1:0/454352144 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6448065fd0 con 0x7f64480fdcf0 2024-06-18T02:40:50.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:50.037+0000 7f6436ffd640 1 -- 172.21.15.1:0/454352144 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7f643c030020 con 0x7f64480fdcf0 2024-06-18T02:40:50.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:50.037+0000 7f6436ffd640 1 --2- 172.21.15.1:0/454352144 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f6430065430 0x7f64300678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:40:50.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:50.037+0000 7f6436ffd640 1 -- 172.21.15.1:0/454352144 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f643c0a2220 con 0x7f64480fdcf0 2024-06-18T02:40:50.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:50.037+0000 7f644659c640 1 --2- 172.21.15.1:0/454352144 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f6430065430 0x7f64300678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:40:50.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:50.038+0000 7f644659c640 1 --2- 172.21.15.1:0/454352144 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f6430065430 0x7f64300678e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f64380041d0 tx=0x7f6438004330 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:40:50.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:50.041+0000 7f6436ffd640 1 -- 172.21.15.1:0/454352144 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f643c070800 con 0x7f64480fdcf0 2024-06-18T02:40:50.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:50.253+0000 7f644c9a1640 1 -- 172.21.15.1:0/454352144 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f64481093f0 con 0x7f64480fdcf0 2024-06-18T02:40:50.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:50.253+0000 7f6436ffd640 1 -- 172.21.15.1:0/454352144 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f643c073e20 con 0x7f64480fdcf0 2024-06-18T02:40:50.253 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:40:50.253 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-18T02:40:50.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:50.255+0000 7f644c9a1640 1 -- 172.21.15.1:0/454352144 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f6430065430 msgr2=0x7f64300678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:40:50.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:50.255+0000 7f644c9a1640 1 --2- 172.21.15.1:0/454352144 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f6430065430 0x7f64300678e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f64380041d0 tx=0x7f6438004330 comp rx=0 tx=0).stop 2024-06-18T02:40:50.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:50.255+0000 7f644c9a1640 1 -- 172.21.15.1:0/454352144 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64480fdcf0 msgr2=0x7f6448115800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:40:50.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:50.255+0000 7f644c9a1640 1 --2- 172.21.15.1:0/454352144 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64480fdcf0 0x7f6448115800 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f643c0048a0 tx=0x7f643c009da0 comp rx=0 tx=0).stop 2024-06-18T02:40:50.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:50.255+0000 7f644c9a1640 1 -- 172.21.15.1:0/454352144 shutdown_connections 2024-06-18T02:40:50.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:50.255+0000 7f644c9a1640 1 --2- 172.21.15.1:0/454352144 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f6430065430 0x7f64300678e0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:40:50.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:50.255+0000 7f644c9a1640 1 --2- 172.21.15.1:0/454352144 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64480fdcf0 0x7f6448115800 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:40:50.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:50.255+0000 7f644c9a1640 1 --2- 172.21.15.1:0/454352144 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64480fd350 0x7f64481152c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:40:50.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:50.255+0000 7f644c9a1640 1 -- 172.21.15.1:0/454352144 >> 172.21.15.1:0/454352144 conn(0x7f64480f6db0 msgr2=0x7f64480f8290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:40:50.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:50.255+0000 7f644c9a1640 1 -- 172.21.15.1:0/454352144 shutdown_connections 2024-06-18T02:40:50.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:50.255+0000 7f644c9a1640 1 -- 172.21.15.1:0/454352144 wait complete. 2024-06-18T02:40:50.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:40:50 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/454352144' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:40:50.987 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:40:50 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/454352144' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:40:51.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:40:51 smithi001 ceph-mon[29099]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:40:51.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:40:51 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:40:51.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:40:51 smithi028 ceph-mon[35823]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:40:51.988 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:40:51 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:40:53.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:40:53 smithi001 ceph-mon[29099]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:40:53.929 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T02:40:54.207 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:40:54.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:40:53 smithi028 ceph-mon[35823]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:40:55.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:55.906+0000 7f7bc1ad2640 1 -- 172.21.15.1:0/4280475586 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bbc101cb0 msgr2=0x7f7bbc0fb230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:40:55.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:55.906+0000 7f7bc1ad2640 1 --2- 172.21.15.1:0/4280475586 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bbc101cb0 0x7f7bbc0fb230 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f7bb00048d0 tx=0x7f7bb0009da0 comp rx=0 tx=0).stop 2024-06-18T02:40:55.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:55.906+0000 7f7bc1ad2640 1 -- 172.21.15.1:0/4280475586 shutdown_connections 2024-06-18T02:40:55.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:55.906+0000 7f7bc1ad2640 1 --2- 172.21.15.1:0/4280475586 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bbc101cb0 0x7f7bbc0fb230 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:40:55.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:55.906+0000 7f7bc1ad2640 1 --2- 172.21.15.1:0/4280475586 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bbc101310 0x7f7bbc1016e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:40:55.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:55.906+0000 7f7bc1ad2640 1 -- 172.21.15.1:0/4280475586 >> 172.21.15.1:0/4280475586 conn(0x7f7bbc0f6d90 msgr2=0x7f7bbc0f91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:40:55.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:55.906+0000 7f7bc1ad2640 1 -- 172.21.15.1:0/4280475586 shutdown_connections 2024-06-18T02:40:55.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:55.906+0000 7f7bc1ad2640 1 -- 172.21.15.1:0/4280475586 wait complete. 2024-06-18T02:40:55.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:55.907+0000 7f7bc1ad2640 1 Processor -- start 2024-06-18T02:40:55.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:55.907+0000 7f7bc1ad2640 1 -- start start 2024-06-18T02:40:55.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:55.907+0000 7f7bc1ad2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bbc101310 0x7f7bbc10e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:40:55.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:55.907+0000 7f7bc1ad2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bbc101cb0 0x7f7bbc10e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:40:55.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:55.907+0000 7f7bc1ad2640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bbc10ec80 con 0x7f7bbc101310 2024-06-18T02:40:55.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:55.907+0000 7f7bc1ad2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bbc10edf0 con 0x7f7bbc101cb0 2024-06-18T02:40:55.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:55.907+0000 7f7bbb7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bbc101310 0x7f7bbc10e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:40:55.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:55.907+0000 7f7bbb7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bbc101310 0x7f7bbc10e0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56202/0 (socket says 172.21.15.1:56202) 2024-06-18T02:40:55.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:55.907+0000 7f7bbb7fe640 1 -- 172.21.15.1:0/2138561102 learned_addr learned my addr 172.21.15.1:0/2138561102 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:40:55.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:55.908+0000 7f7bbb7fe640 1 -- 172.21.15.1:0/2138561102 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bbc101cb0 msgr2=0x7f7bbc10e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:40:55.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:55.908+0000 7f7bbaffd640 1 --2- 172.21.15.1:0/2138561102 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bbc101cb0 0x7f7bbc10e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:40:55.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:55.908+0000 7f7bbb7fe640 1 --2- 172.21.15.1:0/2138561102 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bbc101cb0 0x7f7bbc10e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:40:55.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:55.908+0000 7f7bbb7fe640 1 -- 172.21.15.1:0/2138561102 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bb0005040 con 0x7f7bbc101310 2024-06-18T02:40:55.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:55.908+0000 7f7bbb7fe640 1 --2- 172.21.15.1:0/2138561102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bbc101310 0x7f7bbc10e0f0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f7bac00c790 tx=0x7f7bac0043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:40:55.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:55.908+0000 7f7ba3fff640 1 -- 172.21.15.1:0/2138561102 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7bac00a820 con 0x7f7bbc101310 2024-06-18T02:40:55.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:55.908+0000 7f7ba3fff640 1 -- 172.21.15.1:0/2138561102 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7bac00ae50 con 0x7f7bbc101310 2024-06-18T02:40:55.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:55.908+0000 7f7bc1ad2640 1 -- 172.21.15.1:0/2138561102 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bbc117160 con 0x7f7bbc101310 2024-06-18T02:40:55.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:55.908+0000 7f7ba3fff640 1 -- 172.21.15.1:0/2138561102 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7bac0115e0 con 0x7f7bbc101310 2024-06-18T02:40:55.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:55.908+0000 7f7bc1ad2640 1 -- 172.21.15.1:0/2138561102 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bbc117670 con 0x7f7bbc101310 2024-06-18T02:40:55.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:55.909+0000 7f7bc1ad2640 1 -- 172.21.15.1:0/2138561102 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bbc0fa9f0 con 0x7f7bbc101310 2024-06-18T02:40:55.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:55.910+0000 7f7ba3fff640 1 -- 172.21.15.1:0/2138561102 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7f7bac012400 con 0x7f7bbc101310 2024-06-18T02:40:55.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:55.910+0000 7f7ba3fff640 1 --2- 172.21.15.1:0/2138561102 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f7ba4065430 0x7f7ba40678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:40:55.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:55.910+0000 7f7ba3fff640 1 -- 172.21.15.1:0/2138561102 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f7bac082530 con 0x7f7bbc101310 2024-06-18T02:40:55.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:55.910+0000 7f7bbaffd640 1 --2- 172.21.15.1:0/2138561102 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f7ba4065430 0x7f7ba40678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:40:55.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:55.911+0000 7f7bbaffd640 1 --2- 172.21.15.1:0/2138561102 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f7ba4065430 0x7f7ba40678e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f7bb00047f0 tx=0x7f7bb0009da0 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:40:55.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:55.915+0000 7f7ba3fff640 1 -- 172.21.15.1:0/2138561102 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f7bac050b00 con 0x7f7bbc101310 2024-06-18T02:40:56.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:56.125+0000 7f7bc1ad2640 1 -- 172.21.15.1:0/2138561102 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f7bbc108470 con 0x7f7bbc101310 2024-06-18T02:40:56.125 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:40:55 smithi001 ceph-mon[29099]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:40:56.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:56.125+0000 7f7ba3fff640 1 -- 172.21.15.1:0/2138561102 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f7bac054120 con 0x7f7bbc101310 2024-06-18T02:40:56.126 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:40:56.126 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-18T02:40:56.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:56.128+0000 7f7bc1ad2640 1 -- 172.21.15.1:0/2138561102 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f7ba4065430 msgr2=0x7f7ba40678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:40:56.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:56.128+0000 7f7bc1ad2640 1 --2- 172.21.15.1:0/2138561102 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f7ba4065430 0x7f7ba40678e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f7bb00047f0 tx=0x7f7bb0009da0 comp rx=0 tx=0).stop 2024-06-18T02:40:56.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:56.128+0000 7f7bc1ad2640 1 -- 172.21.15.1:0/2138561102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bbc101310 msgr2=0x7f7bbc10e0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:40:56.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:56.128+0000 7f7bc1ad2640 1 --2- 172.21.15.1:0/2138561102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bbc101310 0x7f7bbc10e0f0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f7bac00c790 tx=0x7f7bac0043c0 comp rx=0 tx=0).stop 2024-06-18T02:40:56.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:56.128+0000 7f7bc1ad2640 1 -- 172.21.15.1:0/2138561102 shutdown_connections 2024-06-18T02:40:56.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:56.128+0000 7f7bc1ad2640 1 --2- 172.21.15.1:0/2138561102 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f7ba4065430 0x7f7ba40678e0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:40:56.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:56.128+0000 7f7bc1ad2640 1 --2- 172.21.15.1:0/2138561102 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bbc101cb0 0x7f7bbc10e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:40:56.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:56.128+0000 7f7bc1ad2640 1 --2- 172.21.15.1:0/2138561102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7bbc101310 0x7f7bbc10e0f0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:40:56.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:56.128+0000 7f7bc1ad2640 1 -- 172.21.15.1:0/2138561102 >> 172.21.15.1:0/2138561102 conn(0x7f7bbc0f6d90 msgr2=0x7f7bbc0f7df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:40:56.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:56.128+0000 7f7bc1ad2640 1 -- 172.21.15.1:0/2138561102 shutdown_connections 2024-06-18T02:40:56.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:40:56.129+0000 7f7bc1ad2640 1 -- 172.21.15.1:0/2138561102 wait complete. 2024-06-18T02:40:56.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:40:55 smithi028 ceph-mon[35823]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:40:57.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:40:56 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/2138561102' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:40:57.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:40:56 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/2138561102' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:40:58.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:40:57 smithi001 ceph-mon[29099]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:40:58.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:40:57 smithi028 ceph-mon[35823]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:40:59.773 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T02:41:00.049 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:41:00.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:40:59 smithi001 ceph-mon[29099]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:41:00.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:40:59 smithi028 ceph-mon[35823]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:41:01.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:01.653+0000 7f32c7e4d640 1 -- 172.21.15.1:0/574814970 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32c0102c40 msgr2=0x7f32c00fc2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:41:01.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:01.653+0000 7f32c7e4d640 1 --2- 172.21.15.1:0/574814970 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32c0102c40 0x7f32c00fc2d0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f32b40048d0 tx=0x7f32b4009da0 comp rx=0 tx=0).stop 2024-06-18T02:41:01.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:01.653+0000 7f32c7e4d640 1 -- 172.21.15.1:0/574814970 shutdown_connections 2024-06-18T02:41:01.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:01.653+0000 7f32c7e4d640 1 --2- 172.21.15.1:0/574814970 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32c0102c40 0x7f32c00fc2d0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:41:01.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:01.653+0000 7f32c7e4d640 1 --2- 172.21.15.1:0/574814970 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32c01022a0 0x7f32c0102670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:41:01.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:01.653+0000 7f32c7e4d640 1 -- 172.21.15.1:0/574814970 >> 172.21.15.1:0/574814970 conn(0x7f32c00f7e70 msgr2=0x7f32c00fa280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:41:01.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:01.653+0000 7f32c7e4d640 1 -- 172.21.15.1:0/574814970 shutdown_connections 2024-06-18T02:41:01.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:01.653+0000 7f32c7e4d640 1 -- 172.21.15.1:0/574814970 wait complete. 2024-06-18T02:41:01.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:01.654+0000 7f32c7e4d640 1 Processor -- start 2024-06-18T02:41:01.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:01.654+0000 7f32c7e4d640 1 -- start start 2024-06-18T02:41:01.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:01.654+0000 7f32c7e4d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32c01022a0 0x7f32c006f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:41:01.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:01.654+0000 7f32c7e4d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32c0102c40 0x7f32c006fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:41:01.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:01.654+0000 7f32c7e4d640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32c00701f0 con 0x7f32c01022a0 2024-06-18T02:41:01.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:01.654+0000 7f32c7e4d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32c0070360 con 0x7f32c0102c40 2024-06-18T02:41:01.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:01.654+0000 7f32c5be9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32c01022a0 0x7f32c006f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:41:01.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:01.654+0000 7f32c5be9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32c01022a0 0x7f32c006f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56212/0 (socket says 172.21.15.1:56212) 2024-06-18T02:41:01.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:01.654+0000 7f32c5be9640 1 -- 172.21.15.1:0/114514864 learned_addr learned my addr 172.21.15.1:0/114514864 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:41:01.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:01.654+0000 7f32c53e8640 1 --2- 172.21.15.1:0/114514864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32c0102c40 0x7f32c006fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:41:01.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:01.654+0000 7f32c5be9640 1 -- 172.21.15.1:0/114514864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32c0102c40 msgr2=0x7f32c006fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:41:01.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:01.654+0000 7f32c5be9640 1 --2- 172.21.15.1:0/114514864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32c0102c40 0x7f32c006fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:41:01.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:01.654+0000 7f32c5be9640 1 -- 172.21.15.1:0/114514864 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32b4005040 con 0x7f32c01022a0 2024-06-18T02:41:01.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:01.654+0000 7f32c5be9640 1 --2- 172.21.15.1:0/114514864 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32c01022a0 0x7f32c006f770 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f32b000db00 tx=0x7f32b0004e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:41:01.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:01.655+0000 7f32be7fc640 1 -- 172.21.15.1:0/114514864 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f32b0017070 con 0x7f32c01022a0 2024-06-18T02:41:01.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:01.655+0000 7f32be7fc640 1 -- 172.21.15.1:0/114514864 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f32b000f040 con 0x7f32c01022a0 2024-06-18T02:41:01.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:01.655+0000 7f32c7e4d640 1 -- 172.21.15.1:0/114514864 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32c00705f0 con 0x7f32c01022a0 2024-06-18T02:41:01.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:01.655+0000 7f32be7fc640 1 -- 172.21.15.1:0/114514864 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f32b0003ae0 con 0x7f32c01022a0 2024-06-18T02:41:01.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:01.655+0000 7f32c7e4d640 1 -- 172.21.15.1:0/114514864 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32c0119760 con 0x7f32c01022a0 2024-06-18T02:41:01.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:01.656+0000 7f32c7e4d640 1 -- 172.21.15.1:0/114514864 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32c00fda10 con 0x7f32c01022a0 2024-06-18T02:41:01.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:01.656+0000 7f32be7fc640 1 -- 172.21.15.1:0/114514864 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7f32b0003c40 con 0x7f32c01022a0 2024-06-18T02:41:01.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:01.656+0000 7f32be7fc640 1 --2- 172.21.15.1:0/114514864 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f32a8065430 0x7f32a80678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:41:01.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:01.657+0000 7f32be7fc640 1 -- 172.21.15.1:0/114514864 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f32b0082600 con 0x7f32c01022a0 2024-06-18T02:41:01.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:01.657+0000 7f32c53e8640 1 --2- 172.21.15.1:0/114514864 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f32a8065430 0x7f32a80678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:41:01.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:01.657+0000 7f32c53e8640 1 --2- 172.21.15.1:0/114514864 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f32a8065430 0x7f32a80678e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f32b4009f80 tx=0x7f32b4035040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:41:01.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:01.661+0000 7f32be7fc640 1 -- 172.21.15.1:0/114514864 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f32b0050a90 con 0x7f32c01022a0 2024-06-18T02:41:01.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:01.875+0000 7f32c7e4d640 1 -- 172.21.15.1:0/114514864 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f32c00fba90 con 0x7f32c01022a0 2024-06-18T02:41:01.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:01.875+0000 7f32be7fc640 1 -- 172.21.15.1:0/114514864 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f32b00540b0 con 0x7f32c01022a0 2024-06-18T02:41:01.875 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:41:01.875 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-18T02:41:01.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:01.877+0000 7f32c7e4d640 1 -- 172.21.15.1:0/114514864 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f32a8065430 msgr2=0x7f32a80678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:41:01.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:01.877+0000 7f32c7e4d640 1 --2- 172.21.15.1:0/114514864 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f32a8065430 0x7f32a80678e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f32b4009f80 tx=0x7f32b4035040 comp rx=0 tx=0).stop 2024-06-18T02:41:01.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:01.877+0000 7f32c7e4d640 1 -- 172.21.15.1:0/114514864 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32c01022a0 msgr2=0x7f32c006f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:41:01.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:01.877+0000 7f32c7e4d640 1 --2- 172.21.15.1:0/114514864 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32c01022a0 0x7f32c006f770 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f32b000db00 tx=0x7f32b0004e60 comp rx=0 tx=0).stop 2024-06-18T02:41:01.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:01.877+0000 7f32c7e4d640 1 -- 172.21.15.1:0/114514864 shutdown_connections 2024-06-18T02:41:01.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:01.877+0000 7f32c7e4d640 1 --2- 172.21.15.1:0/114514864 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f32a8065430 0x7f32a80678e0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:41:01.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:01.877+0000 7f32c7e4d640 1 --2- 172.21.15.1:0/114514864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32c0102c40 0x7f32c006fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:41:01.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:01.877+0000 7f32c7e4d640 1 --2- 172.21.15.1:0/114514864 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32c01022a0 0x7f32c006f770 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:41:01.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:01.877+0000 7f32c7e4d640 1 -- 172.21.15.1:0/114514864 >> 172.21.15.1:0/114514864 conn(0x7f32c00f7e70 msgr2=0x7f32c00f8e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:41:01.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:01.877+0000 7f32c7e4d640 1 -- 172.21.15.1:0/114514864 shutdown_connections 2024-06-18T02:41:01.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:01.877+0000 7f32c7e4d640 1 -- 172.21.15.1:0/114514864 wait complete. 2024-06-18T02:41:02.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:41:01 smithi001 ceph-mon[29099]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:41:02.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:41:01 smithi028 ceph-mon[35823]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:41:02.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:41:02 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/114514864' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:41:03.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:41:02 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/114514864' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:41:03.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:41:03 smithi001 ceph-mon[29099]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:41:04.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:41:03 smithi028 ceph-mon[35823]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:41:05.566 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T02:41:05.844 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:41:05.877 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:41:05 smithi001 ceph-mon[29099]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:41:06.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:41:05 smithi028 ceph-mon[35823]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:41:07.083 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:41:06 smithi001 ceph-mon[29099]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:41:07.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:41:06 smithi028 ceph-mon[35823]: from='mgr.14216 172.21.15.1:0/2828951795' entity='mgr.smithi001.teoawz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:41:07.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:07.604+0000 7f5219271640 1 -- 172.21.15.1:0/2116806012 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f521406fd50 msgr2=0x7f52140701c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:41:07.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:07.604+0000 7f5219271640 1 --2- 172.21.15.1:0/2116806012 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f521406fd50 0x7f52140701c0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f51fc0048d0 tx=0x7f51fc009da0 comp rx=0 tx=0).stop 2024-06-18T02:41:07.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:07.604+0000 7f5219271640 1 -- 172.21.15.1:0/2116806012 shutdown_connections 2024-06-18T02:41:07.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:07.604+0000 7f5219271640 1 --2- 172.21.15.1:0/2116806012 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f521406fd50 0x7f52140701c0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:41:07.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:07.604+0000 7f5219271640 1 --2- 172.21.15.1:0/2116806012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5214071440 0x7f521406f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:41:07.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:07.604+0000 7f5219271640 1 -- 172.21.15.1:0/2116806012 >> 172.21.15.1:0/2116806012 conn(0x7f52140f7f00 msgr2=0x7f52140fa310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:41:07.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:07.604+0000 7f5219271640 1 -- 172.21.15.1:0/2116806012 shutdown_connections 2024-06-18T02:41:07.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:07.604+0000 7f5219271640 1 -- 172.21.15.1:0/2116806012 wait complete. 2024-06-18T02:41:07.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:07.605+0000 7f5219271640 1 Processor -- start 2024-06-18T02:41:07.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:07.605+0000 7f5219271640 1 -- start start 2024-06-18T02:41:07.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:07.605+0000 7f5219271640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f521406fd50 0x7f5214119670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:41:07.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:07.605+0000 7f5219271640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5214071440 0x7f5214119bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:41:07.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:07.605+0000 7f5219271640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f521411a220 con 0x7f5214071440 2024-06-18T02:41:07.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:07.605+0000 7f5219271640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5214113610 con 0x7f521406fd50 2024-06-18T02:41:07.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:07.605+0000 7f520bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5214071440 0x7f5214119bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:41:07.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:07.605+0000 7f520bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5214071440 0x7f5214119bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36446/0 (socket says 172.21.15.1:36446) 2024-06-18T02:41:07.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:07.605+0000 7f520bfff640 1 -- 172.21.15.1:0/451900843 learned_addr learned my addr 172.21.15.1:0/451900843 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:41:07.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:07.605+0000 7f5212ffd640 1 --2- 172.21.15.1:0/451900843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f521406fd50 0x7f5214119670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:41:07.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:07.606+0000 7f520bfff640 1 -- 172.21.15.1:0/451900843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f521406fd50 msgr2=0x7f5214119670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:41:07.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:07.606+0000 7f520bfff640 1 --2- 172.21.15.1:0/451900843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f521406fd50 0x7f5214119670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:41:07.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:07.606+0000 7f520bfff640 1 -- 172.21.15.1:0/451900843 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51fc005040 con 0x7f5214071440 2024-06-18T02:41:07.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:07.606+0000 7f5212ffd640 1 --2- 172.21.15.1:0/451900843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f521406fd50 0x7f5214119670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-18T02:41:07.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:07.606+0000 7f520bfff640 1 --2- 172.21.15.1:0/451900843 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5214071440 0x7f5214119bd0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f51fc003080 tx=0x7f51fc009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:41:07.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:07.606+0000 7f520b7fe640 1 -- 172.21.15.1:0/451900843 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f51fc038070 con 0x7f5214071440 2024-06-18T02:41:07.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:07.606+0000 7f520b7fe640 1 -- 172.21.15.1:0/451900843 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f51fc032e00 con 0x7f5214071440 2024-06-18T02:41:07.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:07.606+0000 7f5219271640 1 -- 172.21.15.1:0/451900843 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5214113890 con 0x7f5214071440 2024-06-18T02:41:07.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:07.606+0000 7f520b7fe640 1 -- 172.21.15.1:0/451900843 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f51fc03de80 con 0x7f5214071440 2024-06-18T02:41:07.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:07.606+0000 7f5219271640 1 -- 172.21.15.1:0/451900843 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5214113d50 con 0x7f5214071440 2024-06-18T02:41:07.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:07.607+0000 7f5219271640 1 -- 172.21.15.1:0/451900843 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5214070f50 con 0x7f5214071440 2024-06-18T02:41:07.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:07.608+0000 7f520b7fe640 1 -- 172.21.15.1:0/451900843 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7f51fc030020 con 0x7f5214071440 2024-06-18T02:41:07.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:07.608+0000 7f520b7fe640 1 --2- 172.21.15.1:0/451900843 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f5200065430 0x7f52000678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:41:07.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:07.608+0000 7f5212ffd640 1 --2- 172.21.15.1:0/451900843 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f5200065430 0x7f52000678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:41:07.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:07.608+0000 7f520b7fe640 1 -- 172.21.15.1:0/451900843 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f51fc0a3140 con 0x7f5214071440 2024-06-18T02:41:07.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:07.609+0000 7f5212ffd640 1 --2- 172.21.15.1:0/451900843 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f5200065430 0x7f52000678e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f5214109320 tx=0x7f5204009040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:41:07.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:07.613+0000 7f520b7fe640 1 -- 172.21.15.1:0/451900843 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f51fc071720 con 0x7f5214071440 2024-06-18T02:41:07.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:07.824+0000 7f5219271640 1 -- 172.21.15.1:0/451900843 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f52140fbb00 con 0x7f5214071440 2024-06-18T02:41:07.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:07.824+0000 7f520b7fe640 1 -- 172.21.15.1:0/451900843 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f51fc074d40 con 0x7f5214071440 2024-06-18T02:41:07.824 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:41:07.824 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-18T02:41:07.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:07.826+0000 7f5219271640 1 -- 172.21.15.1:0/451900843 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f5200065430 msgr2=0x7f52000678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:41:07.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:07.826+0000 7f5219271640 1 --2- 172.21.15.1:0/451900843 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f5200065430 0x7f52000678e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f5214109320 tx=0x7f5204009040 comp rx=0 tx=0).stop 2024-06-18T02:41:07.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:07.826+0000 7f5219271640 1 -- 172.21.15.1:0/451900843 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5214071440 msgr2=0x7f5214119bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:41:07.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:07.826+0000 7f5219271640 1 --2- 172.21.15.1:0/451900843 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5214071440 0x7f5214119bd0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f51fc003080 tx=0x7f51fc009da0 comp rx=0 tx=0).stop 2024-06-18T02:41:07.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:07.826+0000 7f5219271640 1 -- 172.21.15.1:0/451900843 shutdown_connections 2024-06-18T02:41:07.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:07.826+0000 7f5219271640 1 --2- 172.21.15.1:0/451900843 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f5200065430 0x7f52000678e0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:41:07.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:07.826+0000 7f5219271640 1 --2- 172.21.15.1:0/451900843 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5214071440 0x7f5214119bd0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:41:07.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:07.826+0000 7f5219271640 1 --2- 172.21.15.1:0/451900843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f521406fd50 0x7f5214119670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:41:07.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:07.826+0000 7f5219271640 1 -- 172.21.15.1:0/451900843 >> 172.21.15.1:0/451900843 conn(0x7f52140f7f00 msgr2=0x7f52140f90a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:41:07.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:07.826+0000 7f5219271640 1 -- 172.21.15.1:0/451900843 shutdown_connections 2024-06-18T02:41:07.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:07.826+0000 7f5219271640 1 -- 172.21.15.1:0/451900843 wait complete. 2024-06-18T02:41:08.127 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:41:07 smithi001 ceph-mon[29099]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:41:08.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:41:07 smithi028 ceph-mon[35823]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:41:09.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:41:08 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/451900843' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:41:09.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:41:08 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/451900843' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:41:10.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:41:09 smithi001 ceph-mon[29099]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:41:10.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:41:09 smithi028 ceph-mon[35823]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:41:11.515 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T02:41:11.792 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:41:11.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:41:11 smithi001 ceph-mon[29099]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:41:12.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:41:11 smithi028 ceph-mon[35823]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:41:13.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:13.499+0000 7f17fdb6d640 1 -- 172.21.15.1:0/311384621 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17f80fd100 msgr2=0x7f17f80fd4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:41:13.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:13.499+0000 7f17fdb6d640 1 --2- 172.21.15.1:0/311384621 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17f80fd100 0x7f17f80fd4d0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f17e80048d0 tx=0x7f17e8009da0 comp rx=0 tx=0).stop 2024-06-18T02:41:13.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:13.499+0000 7f17fdb6d640 1 -- 172.21.15.1:0/311384621 shutdown_connections 2024-06-18T02:41:13.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:13.499+0000 7f17fdb6d640 1 --2- 172.21.15.1:0/311384621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17f80fa9e0 0x7f17f80fae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:41:13.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:13.499+0000 7f17fdb6d640 1 --2- 172.21.15.1:0/311384621 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17f80fd100 0x7f17f80fd4d0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:41:13.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:13.499+0000 7f17fdb6d640 1 -- 172.21.15.1:0/311384621 >> 172.21.15.1:0/311384621 conn(0x7f17f8070a40 msgr2=0x7f17f8070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:41:13.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:13.499+0000 7f17fdb6d640 1 -- 172.21.15.1:0/311384621 shutdown_connections 2024-06-18T02:41:13.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:13.499+0000 7f17fdb6d640 1 -- 172.21.15.1:0/311384621 wait complete. 2024-06-18T02:41:13.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:13.500+0000 7f17fdb6d640 1 Processor -- start 2024-06-18T02:41:13.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:13.500+0000 7f17fdb6d640 1 -- start start 2024-06-18T02:41:13.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:13.500+0000 7f17fdb6d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17f80fa9e0 0x7f17f81152f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:41:13.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:13.500+0000 7f17fdb6d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17f80fd100 0x7f17f8115830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:41:13.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:13.500+0000 7f17fdb6d640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f8115ec0 con 0x7f17f80fd100 2024-06-18T02:41:13.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:13.500+0000 7f17fdb6d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f810f270 con 0x7f17f80fa9e0 2024-06-18T02:41:13.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:13.500+0000 7f17f6ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17f80fd100 0x7f17f8115830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:41:13.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:13.500+0000 7f17f6ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17f80fd100 0x7f17f8115830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36468/0 (socket says 172.21.15.1:36468) 2024-06-18T02:41:13.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:13.500+0000 7f17f6ffd640 1 -- 172.21.15.1:0/689836369 learned_addr learned my addr 172.21.15.1:0/689836369 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:41:13.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:13.501+0000 7f17f77fe640 1 --2- 172.21.15.1:0/689836369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17f80fa9e0 0x7f17f81152f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:41:13.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:13.501+0000 7f17f6ffd640 1 -- 172.21.15.1:0/689836369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17f80fa9e0 msgr2=0x7f17f81152f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:41:13.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:13.501+0000 7f17f6ffd640 1 --2- 172.21.15.1:0/689836369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17f80fa9e0 0x7f17f81152f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:41:13.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:13.501+0000 7f17f6ffd640 1 -- 172.21.15.1:0/689836369 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17e8005040 con 0x7f17f80fd100 2024-06-18T02:41:13.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:13.501+0000 7f17f6ffd640 1 --2- 172.21.15.1:0/689836369 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17f80fd100 0x7f17f8115830 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f17e000b790 tx=0x7f17e00091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:41:13.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:13.501+0000 7f17effff640 1 -- 172.21.15.1:0/689836369 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f17e000c700 con 0x7f17f80fd100 2024-06-18T02:41:13.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:13.501+0000 7f17fdb6d640 1 -- 172.21.15.1:0/689836369 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17f810f550 con 0x7f17f80fd100 2024-06-18T02:41:13.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:13.501+0000 7f17effff640 1 -- 172.21.15.1:0/689836369 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f17e000cd30 con 0x7f17f80fd100 2024-06-18T02:41:13.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:13.501+0000 7f17effff640 1 -- 172.21.15.1:0/689836369 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f17e0013370 con 0x7f17f80fd100 2024-06-18T02:41:13.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:13.501+0000 7f17fdb6d640 1 -- 172.21.15.1:0/689836369 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17f810fa70 con 0x7f17f80fd100 2024-06-18T02:41:13.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:13.502+0000 7f17fdb6d640 1 -- 172.21.15.1:0/689836369 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17f80fb9f0 con 0x7f17f80fd100 2024-06-18T02:41:13.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:13.503+0000 7f17effff640 1 -- 172.21.15.1:0/689836369 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7f17e0012070 con 0x7f17f80fd100 2024-06-18T02:41:13.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:13.503+0000 7f17effff640 1 --2- 172.21.15.1:0/689836369 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f17e4065430 0x7f17e40678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:41:13.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:13.503+0000 7f17effff640 1 -- 172.21.15.1:0/689836369 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f17e0081dc0 con 0x7f17f80fd100 2024-06-18T02:41:13.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:13.503+0000 7f17f77fe640 1 --2- 172.21.15.1:0/689836369 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f17e4065430 0x7f17e40678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:41:13.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:13.504+0000 7f17f77fe640 1 --2- 172.21.15.1:0/689836369 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f17e4065430 0x7f17e40678e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f17e8009f80 tx=0x7f17e8035040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:41:13.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:13.508+0000 7f17effff640 1 -- 172.21.15.1:0/689836369 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f17e0050270 con 0x7f17f80fd100 2024-06-18T02:41:13.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:13.718+0000 7f17fdb6d640 1 -- 172.21.15.1:0/689836369 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f17f810b6d0 con 0x7f17f80fd100 2024-06-18T02:41:13.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:13.718+0000 7f17effff640 1 -- 172.21.15.1:0/689836369 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f17e0053890 con 0x7f17f80fd100 2024-06-18T02:41:13.718 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:41:13.718 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-18T02:41:13.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:13.720+0000 7f17fdb6d640 1 -- 172.21.15.1:0/689836369 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f17e4065430 msgr2=0x7f17e40678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:41:13.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:13.720+0000 7f17fdb6d640 1 --2- 172.21.15.1:0/689836369 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f17e4065430 0x7f17e40678e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f17e8009f80 tx=0x7f17e8035040 comp rx=0 tx=0).stop 2024-06-18T02:41:13.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:13.720+0000 7f17fdb6d640 1 -- 172.21.15.1:0/689836369 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17f80fd100 msgr2=0x7f17f8115830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:41:13.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:13.720+0000 7f17fdb6d640 1 --2- 172.21.15.1:0/689836369 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17f80fd100 0x7f17f8115830 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f17e000b790 tx=0x7f17e00091d0 comp rx=0 tx=0).stop 2024-06-18T02:41:13.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:13.720+0000 7f17fdb6d640 1 -- 172.21.15.1:0/689836369 shutdown_connections 2024-06-18T02:41:13.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:13.720+0000 7f17fdb6d640 1 --2- 172.21.15.1:0/689836369 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f17e4065430 0x7f17e40678e0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:41:13.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:13.720+0000 7f17fdb6d640 1 --2- 172.21.15.1:0/689836369 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f17f80fd100 0x7f17f8115830 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:41:13.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:13.720+0000 7f17fdb6d640 1 --2- 172.21.15.1:0/689836369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17f80fa9e0 0x7f17f81152f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:41:13.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:13.720+0000 7f17fdb6d640 1 -- 172.21.15.1:0/689836369 >> 172.21.15.1:0/689836369 conn(0x7f17f8070a40 msgr2=0x7f17f80f89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:41:13.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:13.720+0000 7f17fdb6d640 1 -- 172.21.15.1:0/689836369 shutdown_connections 2024-06-18T02:41:13.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:13.720+0000 7f17fdb6d640 1 -- 172.21.15.1:0/689836369 wait complete. 2024-06-18T02:41:13.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:41:13 smithi001 ceph-mon[29099]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:41:13.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:41:13 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/689836369' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:41:14.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:41:13 smithi028 ceph-mon[35823]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:41:14.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:41:13 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/689836369' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:41:16.128 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:41:15 smithi001 ceph-mon[29099]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:41:16.237 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:41:15 smithi028 ceph-mon[35823]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:41:17.440 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T02:41:17.717 INFO:teuthology.orchestra.run.smithi001.stderr:Inferring config /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/mon.smithi001/config 2024-06-18T02:41:18.115 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:41:17 smithi001 ceph-mon[29099]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:41:18.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:41:17 smithi028 ceph-mon[35823]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:41:19.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:19.370+0000 7f94878fb640 1 -- 172.21.15.1:0/4046302897 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9480102c90 msgr2=0x7f94800fc320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:41:19.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:19.370+0000 7f94878fb640 1 --2- 172.21.15.1:0/4046302897 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9480102c90 0x7f94800fc320 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f94700048d0 tx=0x7f9470009da0 comp rx=0 tx=0).stop 2024-06-18T02:41:19.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:19.370+0000 7f94878fb640 1 -- 172.21.15.1:0/4046302897 shutdown_connections 2024-06-18T02:41:19.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:19.370+0000 7f94878fb640 1 --2- 172.21.15.1:0/4046302897 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9480102c90 0x7f94800fc320 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:41:19.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:19.370+0000 7f94878fb640 1 --2- 172.21.15.1:0/4046302897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94801022f0 0x7f94801026c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:41:19.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:19.370+0000 7f94878fb640 1 -- 172.21.15.1:0/4046302897 >> 172.21.15.1:0/4046302897 conn(0x7f94800f7f00 msgr2=0x7f94800fa310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:41:19.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:19.370+0000 7f94878fb640 1 -- 172.21.15.1:0/4046302897 shutdown_connections 2024-06-18T02:41:19.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:19.370+0000 7f94878fb640 1 -- 172.21.15.1:0/4046302897 wait complete. 2024-06-18T02:41:19.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:19.371+0000 7f94878fb640 1 Processor -- start 2024-06-18T02:41:19.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:19.371+0000 7f94878fb640 1 -- start start 2024-06-18T02:41:19.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:19.371+0000 7f94878fb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94801022f0 0x7f948006f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:41:19.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:19.371+0000 7f94878fb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9480102c90 0x7f948006fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:41:19.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:19.371+0000 7f94878fb640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94800702d0 con 0x7f94801022f0 2024-06-18T02:41:19.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:19.371+0000 7f94878fb640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9480070440 con 0x7f9480102c90 2024-06-18T02:41:19.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:19.371+0000 7f9485697640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94801022f0 0x7f948006f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:41:19.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:19.371+0000 7f9485697640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94801022f0 0x7f948006f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36872/0 (socket says 172.21.15.1:36872) 2024-06-18T02:41:19.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:19.371+0000 7f9485697640 1 -- 172.21.15.1:0/659508312 learned_addr learned my addr 172.21.15.1:0/659508312 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-06-18T02:41:19.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:19.371+0000 7f9484e96640 1 --2- 172.21.15.1:0/659508312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9480102c90 0x7f948006fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:41:19.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:19.372+0000 7f9485697640 1 -- 172.21.15.1:0/659508312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9480102c90 msgr2=0x7f948006fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:41:19.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:19.372+0000 7f9485697640 1 --2- 172.21.15.1:0/659508312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9480102c90 0x7f948006fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:41:19.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:19.372+0000 7f9485697640 1 -- 172.21.15.1:0/659508312 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9470005040 con 0x7f94801022f0 2024-06-18T02:41:19.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:19.372+0000 7f9484e96640 1 --2- 172.21.15.1:0/659508312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9480102c90 0x7f948006fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-18T02:41:19.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:19.372+0000 7f9485697640 1 --2- 172.21.15.1:0/659508312 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94801022f0 0x7f948006f770 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f947c00b790 tx=0x7f947c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:41:19.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:19.372+0000 7f9475ffb640 1 -- 172.21.15.1:0/659508312 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f947c00c700 con 0x7f94801022f0 2024-06-18T02:41:19.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:19.372+0000 7f9475ffb640 1 -- 172.21.15.1:0/659508312 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f947c00cd30 con 0x7f94801022f0 2024-06-18T02:41:19.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:19.372+0000 7f94878fb640 1 -- 172.21.15.1:0/659508312 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9480119360 con 0x7f94801022f0 2024-06-18T02:41:19.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:19.372+0000 7f9475ffb640 1 -- 172.21.15.1:0/659508312 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f947c013370 con 0x7f94801022f0 2024-06-18T02:41:19.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:19.372+0000 7f94878fb640 1 -- 172.21.15.1:0/659508312 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9480119820 con 0x7f94801022f0 2024-06-18T02:41:19.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:19.373+0000 7f94878fb640 1 -- 172.21.15.1:0/659508312 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94800fda60 con 0x7f94801022f0 2024-06-18T02:41:19.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:19.373+0000 7f9475ffb640 1 -- 172.21.15.1:0/659508312 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 19) v1 ==== 83743+0+0 (secure 0 0 0) 0x7f947c012070 con 0x7f94801022f0 2024-06-18T02:41:19.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:19.374+0000 7f9475ffb640 1 --2- 172.21.15.1:0/659508312 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f946c065430 0x7f946c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:41:19.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:19.374+0000 7f9475ffb640 1 -- 172.21.15.1:0/659508312 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f947c081fd0 con 0x7f94801022f0 2024-06-18T02:41:19.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:19.374+0000 7f9484e96640 1 --2- 172.21.15.1:0/659508312 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f946c065430 0x7f946c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:41:19.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:19.375+0000 7f9484e96640 1 --2- 172.21.15.1:0/659508312 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f946c065430 0x7f946c0678e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f9470000c00 tx=0x7f9470035040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:41:19.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:19.378+0000 7f9475ffb640 1 -- 172.21.15.1:0/659508312 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f947c050500 con 0x7f94801022f0 2024-06-18T02:41:19.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:19.591+0000 7f94878fb640 1 -- 172.21.15.1:0/659508312 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f94800fbb00 con 0x7f94801022f0 2024-06-18T02:41:19.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:19.591+0000 7f9475ffb640 1 -- 172.21.15.1:0/659508312 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f947c053b20 con 0x7f94801022f0 2024-06-18T02:41:19.591 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-06-18T02:41:19.591 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-18T02:41:19.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:19.593+0000 7f94878fb640 1 -- 172.21.15.1:0/659508312 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f946c065430 msgr2=0x7f946c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:41:19.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:19.593+0000 7f94878fb640 1 --2- 172.21.15.1:0/659508312 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f946c065430 0x7f946c0678e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f9470000c00 tx=0x7f9470035040 comp rx=0 tx=0).stop 2024-06-18T02:41:19.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:19.593+0000 7f94878fb640 1 -- 172.21.15.1:0/659508312 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94801022f0 msgr2=0x7f948006f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:41:19.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:19.593+0000 7f94878fb640 1 --2- 172.21.15.1:0/659508312 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94801022f0 0x7f948006f770 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f947c00b790 tx=0x7f947c0091d0 comp rx=0 tx=0).stop 2024-06-18T02:41:19.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:19.593+0000 7f94878fb640 1 -- 172.21.15.1:0/659508312 shutdown_connections 2024-06-18T02:41:19.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:19.593+0000 7f94878fb640 1 --2- 172.21.15.1:0/659508312 >> [v2:172.21.15.1:6800/729353917,v1:172.21.15.1:6801/729353917] conn(0x7f946c065430 0x7f946c0678e0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:41:19.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:19.593+0000 7f94878fb640 1 --2- 172.21.15.1:0/659508312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9480102c90 0x7f948006fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:41:19.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:19.593+0000 7f94878fb640 1 --2- 172.21.15.1:0/659508312 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f94801022f0 0x7f948006f770 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:41:19.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:19.593+0000 7f94878fb640 1 -- 172.21.15.1:0/659508312 >> 172.21.15.1:0/659508312 conn(0x7f94800f7f00 msgr2=0x7f94800f8e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:41:19.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:19.594+0000 7f94878fb640 1 -- 172.21.15.1:0/659508312 shutdown_connections 2024-06-18T02:41:19.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-06-18T02:41:19.594+0000 7f94878fb640 1 -- 172.21.15.1:0/659508312 wait complete. 2024-06-18T02:41:19.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:41:19 smithi001 ceph-mon[29099]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:41:19.878 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:41:19 smithi001 ceph-mon[29099]: from='client.? 172.21.15.1:0/659508312' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:41:20.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:41:19 smithi028 ceph-mon[35823]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:41:20.238 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:41:19 smithi028 ceph-mon[35823]: from='client.? 172.21.15.1:0/659508312' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:41:20.329 INFO:tasks.cephadm:Teardown begin 2024-06-18T02:41:20.330 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8e9714173de9e92c97e8ef1045d333e96b793454/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph_f18f3f1cbd11e4a41c557e616d172f24deb91473/qa/tasks/cephadm.py", line 2237, in task healthy(ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_ceph_f18f3f1cbd11e4a41c557e616d172f24deb91473/qa/tasks/ceph.py", line 1478, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/git.ceph.com_ceph_f18f3f1cbd11e4a41c557e616d172f24deb91473/qa/tasks/ceph_manager.py", line 3204, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2024-06-18T02:41:20.332 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-18T02:41:20.361 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-18T02:41:20.396 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-06-18T02:41:20.396 DEBUG:teuthology.orchestra.run.smithi001:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-06-18T02:41:20.418 DEBUG:teuthology.orchestra.run.smithi028:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-06-18T02:41:20.454 INFO:tasks.cephadm:Stopping all daemons... 2024-06-18T02:41:20.454 INFO:tasks.cephadm.mon.smithi001:Stopping mon.smithi001... 2024-06-18T02:41:20.455 DEBUG:teuthology.orchestra.run.smithi001:> sudo systemctl stop ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968@mon.smithi001 2024-06-18T02:41:20.628 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:41:20 smithi001 systemd[1]: Stopping Ceph mon.smithi001 for 7eefb63e-2d1a-11ef-bc9d-c7b262605968... 2024-06-18T02:41:21.067 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:41:20 smithi001 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-mon-smithi001[29095]: 2024-06-18T02:41:20.765+0000 7f6bb333c640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi001 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-06-18T02:41:21.067 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:41:20 smithi001 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-mon-smithi001[29095]: 2024-06-18T02:41:20.765+0000 7f6bb333c640 -1 mon.smithi001@0(leader) e2 *** Got Signal Terminated *** 2024-06-18T02:41:21.067 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:41:20 smithi001 podman[80392]: 2024-06-18 02:41:20.897884224 +0000 UTC m=+0.245018292 container died e9c952cac6b572f50bdf1d3bca79e77ed10b21ff653534572a615000cb45c2eb (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-mon-smithi001, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-3fe4105, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240612, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True) 2024-06-18T02:41:21.375 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:41:21 smithi001 podman[80392]: 2024-06-18 02:41:21.257137279 +0000 UTC m=+0.604271349 container cleanup e9c952cac6b572f50bdf1d3bca79e77ed10b21ff653534572a615000cb45c2eb (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-mon-smithi001, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240612, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=quincy-3fe4105, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, io.buildah.version=1.36.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-18T02:41:21.375 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:41:21 smithi001 bash[80392]: ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-mon-smithi001 2024-06-18T02:41:21.628 INFO:journalctl@ceph.mon.smithi001.smithi001.stdout:Jun 18 02:41:21 smithi001 podman[80473]: 2024-06-18 02:41:21.550504237 +0000 UTC m=+0.652829864 container remove e9c952cac6b572f50bdf1d3bca79e77ed10b21ff653534572a615000cb45c2eb (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-mon-smithi001, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=quincy-3fe4105, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240612) 2024-06-18T02:41:21.722 DEBUG:teuthology.orchestra.run.smithi001:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968@mon.smithi001.service' 2024-06-18T02:41:21.773 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-18T02:41:21.773 INFO:tasks.cephadm.mon.smithi001:Stopped mon.smithi001 2024-06-18T02:41:21.773 INFO:tasks.cephadm.mon.smithi028:Stopping mon.smithi028... 2024-06-18T02:41:21.773 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968@mon.smithi028 2024-06-18T02:41:22.085 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:41:21 smithi028 systemd[1]: Stopping Ceph mon.smithi028 for 7eefb63e-2d1a-11ef-bc9d-c7b262605968... 2024-06-18T02:41:22.476 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:41:22 smithi028 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-mon-smithi028[35773]: 2024-06-18T02:41:22.084+0000 7f7d35f62640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi028 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-06-18T02:41:22.477 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:41:22 smithi028 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-mon-smithi028[35773]: 2024-06-18T02:41:22.084+0000 7f7d35f62640 -1 mon.smithi028@1(peon) e2 *** Got Signal Terminated *** 2024-06-18T02:41:22.477 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:41:22 smithi028 podman[59714]: 2024-06-18 02:41:22.147117563 +0000 UTC m=+0.162004029 container died bad1cb26d38b02d096a1d6d9ed53449e22edceefddf511ef58b0eaf35cdfc95c (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-mon-smithi028, io.buildah.version=1.36.0, RELEASE=quincy-3fe4105, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240612) 2024-06-18T02:41:22.737 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:41:22 smithi028 podman[59714]: 2024-06-18 02:41:22.676542572 +0000 UTC m=+0.691429091 container cleanup bad1cb26d38b02d096a1d6d9ed53449e22edceefddf511ef58b0eaf35cdfc95c (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-mon-smithi028, org.label-schema.build-date=20240612, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.36.0, RELEASE=quincy-3fe4105) 2024-06-18T02:41:22.738 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:41:22 smithi028 bash[59714]: ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-mon-smithi028 2024-06-18T02:41:23.106 INFO:journalctl@ceph.mon.smithi028.smithi028.stdout:Jun 18 02:41:22 smithi028 podman[59728]: 2024-06-18 02:41:22.852888777 +0000 UTC m=+0.701987073 container remove bad1cb26d38b02d096a1d6d9ed53449e22edceefddf511ef58b0eaf35cdfc95c (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-mon-smithi028, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=quincy-3fe4105, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.build-date=20240612) 2024-06-18T02:41:23.111 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968@mon.smithi028.service' 2024-06-18T02:41:23.163 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-18T02:41:23.163 INFO:tasks.cephadm.mon.smithi028:Stopped mon.smithi028 2024-06-18T02:41:23.164 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-06-18T02:41:23.164 DEBUG:teuthology.orchestra.run.smithi001:> sudo systemctl stop ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968@osd.0 2024-06-18T02:41:23.487 INFO:journalctl@ceph.osd.0.smithi001.stdout:Jun 18 02:41:23 smithi001 systemd[1]: Stopping Ceph osd.0 for 7eefb63e-2d1a-11ef-bc9d-c7b262605968... 2024-06-18T02:41:23.878 INFO:journalctl@ceph.osd.0.smithi001.stdout:Jun 18 02:41:23 smithi001 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-0[47116]: 2024-06-18T02:41:23.485+0000 7f47e5814640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-18T02:41:23.878 INFO:journalctl@ceph.osd.0.smithi001.stdout:Jun 18 02:41:23 smithi001 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-0[47116]: 2024-06-18T02:41:23.485+0000 7f47e5814640 -1 osd.0 40 *** Got signal Terminated *** 2024-06-18T02:41:23.878 INFO:journalctl@ceph.osd.0.smithi001.stdout:Jun 18 02:41:23 smithi001 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-0[47116]: 2024-06-18T02:41:23.485+0000 7f47e5814640 -1 osd.0 40 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-18T02:41:29.128 INFO:journalctl@ceph.osd.0.smithi001.stdout:Jun 18 02:41:28 smithi001 podman[80624]: 2024-06-18 02:41:28.817902187 +0000 UTC m=+5.444004484 container died 3c09c61df658b61320810cebc08c420da17d61afc3f4758b713e33337e6fc2e6 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-0, org.label-schema.build-date=20240612, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, RELEASE=quincy-3fe4105, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_BRANCH=HEAD) 2024-06-18T02:41:29.571 INFO:journalctl@ceph.osd.0.smithi001.stdout:Jun 18 02:41:29 smithi001 podman[80624]: 2024-06-18 02:41:29.320486783 +0000 UTC m=+5.946589079 container cleanup 3c09c61df658b61320810cebc08c420da17d61afc3f4758b713e33337e6fc2e6 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, org.label-schema.build-date=20240612, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-3fe4105) 2024-06-18T02:41:29.571 INFO:journalctl@ceph.osd.0.smithi001.stdout:Jun 18 02:41:29 smithi001 bash[80624]: ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-0 2024-06-18T02:41:29.878 INFO:journalctl@ceph.osd.0.smithi001.stdout:Jun 18 02:41:29 smithi001 podman[80640]: 2024-06-18 02:41:29.57193016 +0000 UTC m=+0.744721368 container remove 3c09c61df658b61320810cebc08c420da17d61afc3f4758b713e33337e6fc2e6 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-0, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=quincy-3fe4105, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.build-date=20240612, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-06-18T02:41:30.166 INFO:journalctl@ceph.osd.0.smithi001.stdout:Jun 18 02:41:29 smithi001 podman[80727]: 2024-06-18 02:41:29.817929187 +0000 UTC m=+0.018013318 image pull 1a271607156e1b6becab587233eefa647399ff75aa2bb76fdb2f0360ec5b7818 quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032 2024-06-18T02:41:30.166 INFO:journalctl@ceph.osd.0.smithi001.stdout:Jun 18 02:41:29 smithi001 podman[80727]: 2024-06-18 02:41:29.922798921 +0000 UTC m=+0.122883093 container create f1cf5daa88c2833327641440c9fd4377520764cd53d796dc0475cd1c3e168614 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.build-date=20240612, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.schema-version=1.0, RELEASE=quincy-3fe4105, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2) 2024-06-18T02:41:30.166 INFO:journalctl@ceph.osd.0.smithi001.stdout:Jun 18 02:41:30 smithi001 podman[80727]: 2024-06-18 02:41:30.167323198 +0000 UTC m=+0.367407328 container init f1cf5daa88c2833327641440c9fd4377520764cd53d796dc0475cd1c3e168614 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=quincy-3fe4105, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_BRANCH=HEAD, org.label-schema.build-date=20240612) 2024-06-18T02:41:30.427 INFO:journalctl@ceph.osd.0.smithi001.stdout:Jun 18 02:41:30 smithi001 podman[80727]: 2024-06-18 02:41:30.170908897 +0000 UTC m=+0.370993032 container start f1cf5daa88c2833327641440c9fd4377520764cd53d796dc0475cd1c3e168614 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-3fe4105, org.label-schema.build-date=20240612, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-06-18T02:41:30.427 INFO:journalctl@ceph.osd.0.smithi001.stdout:Jun 18 02:41:30 smithi001 podman[80727]: 2024-06-18 02:41:30.242170495 +0000 UTC m=+0.442254643 container attach f1cf5daa88c2833327641440c9fd4377520764cd53d796dc0475cd1c3e168614 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240612, RELEASE=quincy-3fe4105, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-06-18T02:41:30.727 INFO:journalctl@ceph.osd.0.smithi001.stdout:Jun 18 02:41:30 smithi001 podman[80727]: 2024-06-18 02:41:30.428656201 +0000 UTC m=+0.628740339 container died f1cf5daa88c2833327641440c9fd4377520764cd53d796dc0475cd1c3e168614 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240612, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, RELEASE=quincy-3fe4105, ceph=True) 2024-06-18T02:41:31.042 DEBUG:teuthology.orchestra.run.smithi001:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968@osd.0.service' 2024-06-18T02:41:31.087 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-18T02:41:31.087 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-06-18T02:41:31.087 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-06-18T02:41:31.088 DEBUG:teuthology.orchestra.run.smithi001:> sudo systemctl stop ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968@osd.1 2024-06-18T02:41:31.378 INFO:journalctl@ceph.osd.1.smithi001.stdout:Jun 18 02:41:31 smithi001 systemd[1]: Stopping Ceph osd.1 for 7eefb63e-2d1a-11ef-bc9d-c7b262605968... 2024-06-18T02:41:31.878 INFO:journalctl@ceph.osd.1.smithi001.stdout:Jun 18 02:41:31 smithi001 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-1[54605]: 2024-06-18T02:41:31.380+0000 7f3123675640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-18T02:41:31.878 INFO:journalctl@ceph.osd.1.smithi001.stdout:Jun 18 02:41:31 smithi001 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-1[54605]: 2024-06-18T02:41:31.380+0000 7f3123675640 -1 osd.1 40 *** Got signal Terminated *** 2024-06-18T02:41:31.878 INFO:journalctl@ceph.osd.1.smithi001.stdout:Jun 18 02:41:31 smithi001 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-1[54605]: 2024-06-18T02:41:31.380+0000 7f3123675640 -1 osd.1 40 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-18T02:41:37.166 INFO:journalctl@ceph.osd.1.smithi001.stdout:Jun 18 02:41:36 smithi001 podman[80829]: 2024-06-18 02:41:36.89704925 +0000 UTC m=+5.618000635 container died 3dfac1bbc28bb380c67c19ff7ba239d838b3e013ff95494d1ff541d3abb868f8 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-1, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, RELEASE=quincy-3fe4105, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.build-date=20240612, org.label-schema.schema-version=1.0) 2024-06-18T02:41:37.628 INFO:journalctl@ceph.osd.1.smithi001.stdout:Jun 18 02:41:37 smithi001 podman[80829]: 2024-06-18 02:41:37.382038823 +0000 UTC m=+6.102990240 container cleanup 3dfac1bbc28bb380c67c19ff7ba239d838b3e013ff95494d1ff541d3abb868f8 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-1, GIT_CLEAN=True, RELEASE=quincy-3fe4105, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240612, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-06-18T02:41:37.628 INFO:journalctl@ceph.osd.1.smithi001.stdout:Jun 18 02:41:37 smithi001 bash[80829]: ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-1 2024-06-18T02:41:38.014 INFO:journalctl@ceph.osd.1.smithi001.stdout:Jun 18 02:41:37 smithi001 podman[80847]: 2024-06-18 02:41:37.641759909 +0000 UTC m=+0.741489477 container remove 3dfac1bbc28bb380c67c19ff7ba239d838b3e013ff95494d1ff541d3abb868f8 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240612, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, RELEASE=quincy-3fe4105, io.buildah.version=1.36.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-06-18T02:41:38.344 INFO:journalctl@ceph.osd.1.smithi001.stdout:Jun 18 02:41:38 smithi001 podman[80933]: 2024-06-18 02:41:37.917856472 +0000 UTC m=+0.017245130 image pull 1a271607156e1b6becab587233eefa647399ff75aa2bb76fdb2f0360ec5b7818 quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032 2024-06-18T02:41:38.344 INFO:journalctl@ceph.osd.1.smithi001.stdout:Jun 18 02:41:38 smithi001 podman[80933]: 2024-06-18 02:41:38.036597246 +0000 UTC m=+0.135985930 container create 1e040fd8a495d324acd5202bf070a7bb6a0efb4fc88f50dc1833413f6505fd35 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, RELEASE=quincy-3fe4105, io.buildah.version=1.36.0, org.label-schema.build-date=20240612, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-06-18T02:41:38.344 INFO:journalctl@ceph.osd.1.smithi001.stdout:Jun 18 02:41:38 smithi001 podman[80933]: 2024-06-18 02:41:38.261700076 +0000 UTC m=+0.361088723 container init 1e040fd8a495d324acd5202bf070a7bb6a0efb4fc88f50dc1833413f6505fd35 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240612, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=quincy-3fe4105, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-18T02:41:38.344 INFO:journalctl@ceph.osd.1.smithi001.stdout:Jun 18 02:41:38 smithi001 podman[80933]: 2024-06-18 02:41:38.265144359 +0000 UTC m=+0.364533001 container start 1e040fd8a495d324acd5202bf070a7bb6a0efb4fc88f50dc1833413f6505fd35 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-1-deactivate, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-3fe4105, GIT_BRANCH=HEAD, org.label-schema.build-date=20240612, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=) 2024-06-18T02:41:38.344 INFO:journalctl@ceph.osd.1.smithi001.stdout:Jun 18 02:41:38 smithi001 podman[80933]: 2024-06-18 02:41:38.344900576 +0000 UTC m=+0.444289230 container attach 1e040fd8a495d324acd5202bf070a7bb6a0efb4fc88f50dc1833413f6505fd35 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-1-deactivate, org.label-schema.build-date=20240612, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=quincy-3fe4105) 2024-06-18T02:41:38.630 INFO:journalctl@ceph.osd.1.smithi001.stdout:Jun 18 02:41:38 smithi001 podman[80933]: 2024-06-18 02:41:38.519641205 +0000 UTC m=+0.619029855 container died 1e040fd8a495d324acd5202bf070a7bb6a0efb4fc88f50dc1833413f6505fd35 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-1-deactivate, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=quincy-3fe4105, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240612, org.label-schema.vendor=CentOS) 2024-06-18T02:41:39.254 DEBUG:teuthology.orchestra.run.smithi001:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968@osd.1.service' 2024-06-18T02:41:39.297 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-18T02:41:39.297 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-06-18T02:41:39.298 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-06-18T02:41:39.298 DEBUG:teuthology.orchestra.run.smithi001:> sudo systemctl stop ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968@osd.2 2024-06-18T02:41:39.627 INFO:journalctl@ceph.osd.2.smithi001.stdout:Jun 18 02:41:39 smithi001 systemd[1]: Stopping Ceph osd.2 for 7eefb63e-2d1a-11ef-bc9d-c7b262605968... 2024-06-18T02:41:40.128 INFO:journalctl@ceph.osd.2.smithi001.stdout:Jun 18 02:41:39 smithi001 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-2[62339]: 2024-06-18T02:41:39.635+0000 7f664bc78640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-18T02:41:40.128 INFO:journalctl@ceph.osd.2.smithi001.stdout:Jun 18 02:41:39 smithi001 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-2[62339]: 2024-06-18T02:41:39.635+0000 7f664bc78640 -1 osd.2 40 *** Got signal Terminated *** 2024-06-18T02:41:40.128 INFO:journalctl@ceph.osd.2.smithi001.stdout:Jun 18 02:41:39 smithi001 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-2[62339]: 2024-06-18T02:41:39.635+0000 7f664bc78640 -1 osd.2 40 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-18T02:41:45.378 INFO:journalctl@ceph.osd.2.smithi001.stdout:Jun 18 02:41:44 smithi001 podman[81035]: 2024-06-18 02:41:44.978263092 +0000 UTC m=+5.450168410 container died 8993d3a0a5e6cde384cffc2fa3201efc5691b39fb080854d92f2d89481271727 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-2, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, RELEASE=quincy-3fe4105, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240612, org.label-schema.vendor=CentOS) 2024-06-18T02:41:45.844 INFO:journalctl@ceph.osd.2.smithi001.stdout:Jun 18 02:41:45 smithi001 podman[81035]: 2024-06-18 02:41:45.534986635 +0000 UTC m=+6.006891952 container cleanup 8993d3a0a5e6cde384cffc2fa3201efc5691b39fb080854d92f2d89481271727 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-2, org.label-schema.build-date=20240612, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.36.0, RELEASE=quincy-3fe4105) 2024-06-18T02:41:45.844 INFO:journalctl@ceph.osd.2.smithi001.stdout:Jun 18 02:41:45 smithi001 bash[81035]: ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-2 2024-06-18T02:41:46.128 INFO:journalctl@ceph.osd.2.smithi001.stdout:Jun 18 02:41:45 smithi001 podman[81051]: 2024-06-18 02:41:45.844618276 +0000 UTC m=+0.858977236 container remove 8993d3a0a5e6cde384cffc2fa3201efc5691b39fb080854d92f2d89481271727 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-2, RELEASE=quincy-3fe4105, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.build-date=20240612, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47) 2024-06-18T02:41:46.538 INFO:journalctl@ceph.osd.2.smithi001.stdout:Jun 18 02:41:46 smithi001 podman[81140]: 2024-06-18 02:41:46.178624574 +0000 UTC m=+0.114734735 container create 71cb0c90ebf62c63faca28a10dd4666666871c6239516860bbf6c4ad9479d95b (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-2-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240612, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , RELEASE=quincy-3fe4105, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-18T02:41:46.539 INFO:journalctl@ceph.osd.2.smithi001.stdout:Jun 18 02:41:46 smithi001 podman[81140]: 2024-06-18 02:41:46.082268944 +0000 UTC m=+0.018379110 image pull 1a271607156e1b6becab587233eefa647399ff75aa2bb76fdb2f0360ec5b7818 quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032 2024-06-18T02:41:46.539 INFO:journalctl@ceph.osd.2.smithi001.stdout:Jun 18 02:41:46 smithi001 podman[81140]: 2024-06-18 02:41:46.539715968 +0000 UTC m=+0.475826139 container init 71cb0c90ebf62c63faca28a10dd4666666871c6239516860bbf6c4ad9479d95b (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-2-deactivate, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, GIT_BRANCH=HEAD, RELEASE=quincy-3fe4105, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240612, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, ceph=True) 2024-06-18T02:41:46.805 INFO:journalctl@ceph.osd.2.smithi001.stdout:Jun 18 02:41:46 smithi001 podman[81140]: 2024-06-18 02:41:46.543184914 +0000 UTC m=+0.479295071 container start 71cb0c90ebf62c63faca28a10dd4666666871c6239516860bbf6c4ad9479d95b (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-2-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240612, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-3fe4105, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, io.buildah.version=1.36.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47) 2024-06-18T02:41:46.805 INFO:journalctl@ceph.osd.2.smithi001.stdout:Jun 18 02:41:46 smithi001 podman[81140]: 2024-06-18 02:41:46.63115586 +0000 UTC m=+0.567266029 container attach 71cb0c90ebf62c63faca28a10dd4666666871c6239516860bbf6c4ad9479d95b (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-2-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240612, RELEASE=quincy-3fe4105, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True) 2024-06-18T02:41:47.092 INFO:journalctl@ceph.osd.2.smithi001.stdout:Jun 18 02:41:46 smithi001 podman[81140]: 2024-06-18 02:41:46.806360378 +0000 UTC m=+0.742470542 container died 71cb0c90ebf62c63faca28a10dd4666666871c6239516860bbf6c4ad9479d95b (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-2-deactivate, ceph=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240612, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=quincy-3fe4105, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-06-18T02:41:47.373 DEBUG:teuthology.orchestra.run.smithi001:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968@osd.2.service' 2024-06-18T02:41:47.412 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-18T02:41:47.412 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-06-18T02:41:47.413 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-06-18T02:41:47.413 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968@osd.3 2024-06-18T02:41:47.711 INFO:journalctl@ceph.osd.3.smithi028.stdout:Jun 18 02:41:47 smithi028 systemd[1]: Stopping Ceph osd.3 for 7eefb63e-2d1a-11ef-bc9d-c7b262605968... 2024-06-18T02:41:47.988 INFO:journalctl@ceph.osd.3.smithi028.stdout:Jun 18 02:41:47 smithi028 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-3[41170]: 2024-06-18T02:41:47.711+0000 7f103b542640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-18T02:41:47.988 INFO:journalctl@ceph.osd.3.smithi028.stdout:Jun 18 02:41:47 smithi028 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-3[41170]: 2024-06-18T02:41:47.711+0000 7f103b542640 -1 osd.3 40 *** Got signal Terminated *** 2024-06-18T02:41:47.988 INFO:journalctl@ceph.osd.3.smithi028.stdout:Jun 18 02:41:47 smithi028 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-3[41170]: 2024-06-18T02:41:47.711+0000 7f103b542640 -1 osd.3 40 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-18T02:41:51.238 INFO:journalctl@ceph.osd.5.smithi028.stdout:Jun 18 02:41:50 smithi028 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-5[54541]: 2024-06-18T02:41:50.778+0000 7fe438624640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-06-18T02:41:25.341915+0000 front 2024-06-18T02:41:25.342013+0000 (oldest deadline 2024-06-18T02:41:50.041789+0000) 2024-06-18T02:41:52.237 INFO:journalctl@ceph.osd.5.smithi028.stdout:Jun 18 02:41:51 smithi028 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-5[54541]: 2024-06-18T02:41:51.804+0000 7fe438624640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-06-18T02:41:25.341915+0000 front 2024-06-18T02:41:25.342013+0000 (oldest deadline 2024-06-18T02:41:50.041789+0000) 2024-06-18T02:41:53.238 INFO:journalctl@ceph.osd.3.smithi028.stdout:Jun 18 02:41:53 smithi028 podman[59836]: 2024-06-18 02:41:53.070066693 +0000 UTC m=+5.451729061 container died 7f41020875e3e4896b05d29bf3ac6966791a71b0c581ccf7940f9d0a14736b05 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-3, ceph=True, RELEASE=quincy-3fe4105, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240612, org.label-schema.schema-version=1.0) 2024-06-18T02:41:53.238 INFO:journalctl@ceph.osd.5.smithi028.stdout:Jun 18 02:41:52 smithi028 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-5[54541]: 2024-06-18T02:41:52.837+0000 7fe438624640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-06-18T02:41:25.341915+0000 front 2024-06-18T02:41:25.342013+0000 (oldest deadline 2024-06-18T02:41:50.041789+0000) 2024-06-18T02:41:53.687 INFO:journalctl@ceph.osd.4.smithi028.stdout:Jun 18 02:41:53 smithi028 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-4[47983]: 2024-06-18T02:41:53.345+0000 7f8647503640 -1 osd.4 40 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-06-18T02:41:28.430817+0000 front 2024-06-18T02:41:28.430788+0000 (oldest deadline 2024-06-18T02:41:53.130673+0000) 2024-06-18T02:41:53.688 INFO:journalctl@ceph.osd.3.smithi028.stdout:Jun 18 02:41:53 smithi028 podman[59836]: 2024-06-18 02:41:53.43664595 +0000 UTC m=+5.818308321 container cleanup 7f41020875e3e4896b05d29bf3ac6966791a71b0c581ccf7940f9d0a14736b05 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-3, ceph=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240612, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, RELEASE=quincy-3fe4105, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2) 2024-06-18T02:41:53.688 INFO:journalctl@ceph.osd.3.smithi028.stdout:Jun 18 02:41:53 smithi028 bash[59836]: ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-3 2024-06-18T02:41:53.988 INFO:journalctl@ceph.osd.3.smithi028.stdout:Jun 18 02:41:53 smithi028 podman[59863]: 2024-06-18 02:41:53.688129056 +0000 UTC m=+0.607037784 container remove 7f41020875e3e4896b05d29bf3ac6966791a71b0c581ccf7940f9d0a14736b05 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-3, RELEASE=quincy-3fe4105, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240612, io.buildah.version=1.36.0, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-06-18T02:41:53.988 INFO:journalctl@ceph.osd.5.smithi028.stdout:Jun 18 02:41:53 smithi028 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-5[54541]: 2024-06-18T02:41:53.837+0000 7fe438624640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-06-18T02:41:25.341915+0000 front 2024-06-18T02:41:25.342013+0000 (oldest deadline 2024-06-18T02:41:50.041789+0000) 2024-06-18T02:41:54.333 INFO:journalctl@ceph.osd.3.smithi028.stdout:Jun 18 02:41:54 smithi028 podman[59955]: 2024-06-18 02:41:53.928622063 +0000 UTC m=+0.017854710 image pull 1a271607156e1b6becab587233eefa647399ff75aa2bb76fdb2f0360ec5b7818 quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032 2024-06-18T02:41:54.333 INFO:journalctl@ceph.osd.3.smithi028.stdout:Jun 18 02:41:54 smithi028 podman[59955]: 2024-06-18 02:41:54.122565883 +0000 UTC m=+0.211798527 container create f18efae815e920978e21fcbd027804effec45038a26d7d9277c99b2a0a123e0b (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240612, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, RELEASE=quincy-3fe4105) 2024-06-18T02:41:54.333 INFO:journalctl@ceph.osd.3.smithi028.stdout:Jun 18 02:41:54 smithi028 podman[59955]: 2024-06-18 02:41:54.334051557 +0000 UTC m=+0.423284203 container init f18efae815e920978e21fcbd027804effec45038a26d7d9277c99b2a0a123e0b (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-3-deactivate, RELEASE=quincy-3fe4105, org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.build-date=20240612, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-06-18T02:41:54.598 INFO:journalctl@ceph.osd.4.smithi028.stdout:Jun 18 02:41:54 smithi028 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-4[47983]: 2024-06-18T02:41:54.336+0000 7f8647503640 -1 osd.4 40 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-06-18T02:41:28.430817+0000 front 2024-06-18T02:41:28.430788+0000 (oldest deadline 2024-06-18T02:41:53.130673+0000) 2024-06-18T02:41:54.598 INFO:journalctl@ceph.osd.3.smithi028.stdout:Jun 18 02:41:54 smithi028 podman[59955]: 2024-06-18 02:41:54.337564875 +0000 UTC m=+0.426797520 container start f18efae815e920978e21fcbd027804effec45038a26d7d9277c99b2a0a123e0b (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-3-deactivate, ceph=True, org.label-schema.build-date=20240612, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.vendor=CentOS, RELEASE=quincy-3fe4105) 2024-06-18T02:41:54.598 INFO:journalctl@ceph.osd.3.smithi028.stdout:Jun 18 02:41:54 smithi028 podman[59955]: 2024-06-18 02:41:54.409071746 +0000 UTC m=+0.498304396 container attach f18efae815e920978e21fcbd027804effec45038a26d7d9277c99b2a0a123e0b (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-3-deactivate, GIT_CLEAN=True, RELEASE=quincy-3fe4105, org.label-schema.build-date=20240612, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-06-18T02:41:54.902 INFO:journalctl@ceph.osd.3.smithi028.stdout:Jun 18 02:41:54 smithi028 podman[59955]: 2024-06-18 02:41:54.59795273 +0000 UTC m=+0.687185373 container died f18efae815e920978e21fcbd027804effec45038a26d7d9277c99b2a0a123e0b (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.build-date=20240612, RELEASE=quincy-3fe4105, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-06-18T02:41:54.903 INFO:journalctl@ceph.osd.5.smithi028.stdout:Jun 18 02:41:54 smithi028 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-5[54541]: 2024-06-18T02:41:54.812+0000 7fe438624640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-06-18T02:41:25.341915+0000 front 2024-06-18T02:41:25.342013+0000 (oldest deadline 2024-06-18T02:41:50.041789+0000) 2024-06-18T02:41:55.202 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968@osd.3.service' 2024-06-18T02:41:55.242 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-18T02:41:55.243 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-06-18T02:41:55.243 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-06-18T02:41:55.243 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968@osd.4 2024-06-18T02:41:55.487 INFO:journalctl@ceph.osd.4.smithi028.stdout:Jun 18 02:41:55 smithi028 systemd[1]: Stopping Ceph osd.4 for 7eefb63e-2d1a-11ef-bc9d-c7b262605968... 2024-06-18T02:41:55.488 INFO:journalctl@ceph.osd.4.smithi028.stdout:Jun 18 02:41:55 smithi028 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-4[47983]: 2024-06-18T02:41:55.370+0000 7f8647503640 -1 osd.4 40 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-06-18T02:41:28.430817+0000 front 2024-06-18T02:41:28.430788+0000 (oldest deadline 2024-06-18T02:41:53.130673+0000) 2024-06-18T02:41:55.988 INFO:journalctl@ceph.osd.5.smithi028.stdout:Jun 18 02:41:55 smithi028 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-5[54541]: 2024-06-18T02:41:55.809+0000 7fe438624640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-06-18T02:41:25.341915+0000 front 2024-06-18T02:41:25.342013+0000 (oldest deadline 2024-06-18T02:41:50.041789+0000) 2024-06-18T02:41:55.988 INFO:journalctl@ceph.osd.4.smithi028.stdout:Jun 18 02:41:55 smithi028 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-4[47983]: 2024-06-18T02:41:55.656+0000 7f864a6ea640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-18T02:41:55.988 INFO:journalctl@ceph.osd.4.smithi028.stdout:Jun 18 02:41:55 smithi028 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-4[47983]: 2024-06-18T02:41:55.656+0000 7f864a6ea640 -1 osd.4 40 *** Got signal Terminated *** 2024-06-18T02:41:55.988 INFO:journalctl@ceph.osd.4.smithi028.stdout:Jun 18 02:41:55 smithi028 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-4[47983]: 2024-06-18T02:41:55.656+0000 7f864a6ea640 -1 osd.4 40 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-18T02:41:56.738 INFO:journalctl@ceph.osd.4.smithi028.stdout:Jun 18 02:41:56 smithi028 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-4[47983]: 2024-06-18T02:41:56.380+0000 7f8647503640 -1 osd.4 40 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-06-18T02:41:28.430817+0000 front 2024-06-18T02:41:28.430788+0000 (oldest deadline 2024-06-18T02:41:53.130673+0000) 2024-06-18T02:41:57.237 INFO:journalctl@ceph.osd.5.smithi028.stdout:Jun 18 02:41:56 smithi028 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-5[54541]: 2024-06-18T02:41:56.829+0000 7fe438624640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-06-18T02:41:25.341915+0000 front 2024-06-18T02:41:25.342013+0000 (oldest deadline 2024-06-18T02:41:50.041789+0000) 2024-06-18T02:41:57.738 INFO:journalctl@ceph.osd.4.smithi028.stdout:Jun 18 02:41:57 smithi028 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-4[47983]: 2024-06-18T02:41:57.398+0000 7f8647503640 -1 osd.4 40 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-06-18T02:41:28.430817+0000 front 2024-06-18T02:41:28.430788+0000 (oldest deadline 2024-06-18T02:41:53.130673+0000) 2024-06-18T02:41:58.237 INFO:journalctl@ceph.osd.5.smithi028.stdout:Jun 18 02:41:57 smithi028 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-5[54541]: 2024-06-18T02:41:57.810+0000 7fe438624640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-06-18T02:41:25.341915+0000 front 2024-06-18T02:41:25.342013+0000 (oldest deadline 2024-06-18T02:41:50.041789+0000) 2024-06-18T02:41:58.738 INFO:journalctl@ceph.osd.4.smithi028.stdout:Jun 18 02:41:58 smithi028 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-4[47983]: 2024-06-18T02:41:58.386+0000 7f8647503640 -1 osd.4 40 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-06-18T02:41:28.430817+0000 front 2024-06-18T02:41:28.430788+0000 (oldest deadline 2024-06-18T02:41:53.130673+0000) 2024-06-18T02:41:59.238 INFO:journalctl@ceph.osd.5.smithi028.stdout:Jun 18 02:41:58 smithi028 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-5[54541]: 2024-06-18T02:41:58.823+0000 7fe438624640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-06-18T02:41:25.341915+0000 front 2024-06-18T02:41:25.342013+0000 (oldest deadline 2024-06-18T02:41:50.041789+0000) 2024-06-18T02:41:59.738 INFO:journalctl@ceph.osd.4.smithi028.stdout:Jun 18 02:41:59 smithi028 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-4[47983]: 2024-06-18T02:41:59.394+0000 7f8647503640 -1 osd.4 40 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-06-18T02:41:28.430817+0000 front 2024-06-18T02:41:28.430788+0000 (oldest deadline 2024-06-18T02:41:53.130673+0000) 2024-06-18T02:42:00.238 INFO:journalctl@ceph.osd.5.smithi028.stdout:Jun 18 02:41:59 smithi028 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-5[54541]: 2024-06-18T02:41:59.863+0000 7fe438624640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-06-18T02:41:25.341915+0000 front 2024-06-18T02:41:25.342013+0000 (oldest deadline 2024-06-18T02:41:50.041789+0000) 2024-06-18T02:42:00.738 INFO:journalctl@ceph.osd.4.smithi028.stdout:Jun 18 02:42:00 smithi028 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-4[47983]: 2024-06-18T02:42:00.346+0000 7f8647503640 -1 osd.4 40 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-06-18T02:41:28.430817+0000 front 2024-06-18T02:41:28.430788+0000 (oldest deadline 2024-06-18T02:41:53.130673+0000) 2024-06-18T02:42:00.738 INFO:journalctl@ceph.osd.4.smithi028.stdout:Jun 18 02:42:00 smithi028 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-4[47983]: 2024-06-18T02:42:00.346+0000 7f8647503640 -1 osd.4 40 heartbeat_check: no reply from 172.21.15.1:6814 osd.1 since back 2024-06-18T02:41:33.131220+0000 front 2024-06-18T02:41:33.131230+0000 (oldest deadline 2024-06-18T02:42:00.131464+0000) 2024-06-18T02:42:01.115 INFO:journalctl@ceph.osd.5.smithi028.stdout:Jun 18 02:42:00 smithi028 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-5[54541]: 2024-06-18T02:42:00.835+0000 7fe438624640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-06-18T02:41:25.341915+0000 front 2024-06-18T02:41:25.342013+0000 (oldest deadline 2024-06-18T02:41:50.041789+0000) 2024-06-18T02:42:01.390 INFO:journalctl@ceph.osd.4.smithi028.stdout:Jun 18 02:42:01 smithi028 podman[60060]: 2024-06-18 02:42:01.115819809 +0000 UTC m=+5.646884877 container died 4169d0c0b00be73b52f14e7da12f8ec1f07775ea944a6a4cb8ea6b068a6a0523 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-4, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240612, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=quincy-3fe4105, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, io.buildah.version=1.36.0) 2024-06-18T02:42:01.707 INFO:journalctl@ceph.osd.4.smithi028.stdout:Jun 18 02:42:01 smithi028 podman[60060]: 2024-06-18 02:42:01.490433347 +0000 UTC m=+6.021498415 container cleanup 4169d0c0b00be73b52f14e7da12f8ec1f07775ea944a6a4cb8ea6b068a6a0523 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-4, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240612, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-3fe4105, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-06-18T02:42:01.707 INFO:journalctl@ceph.osd.4.smithi028.stdout:Jun 18 02:42:01 smithi028 bash[60060]: ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-4 2024-06-18T02:42:01.978 INFO:journalctl@ceph.osd.4.smithi028.stdout:Jun 18 02:42:01 smithi028 podman[60077]: 2024-06-18 02:42:01.708022678 +0000 UTC m=+0.591055787 container remove 4169d0c0b00be73b52f14e7da12f8ec1f07775ea944a6a4cb8ea6b068a6a0523 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-4, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240612, RELEASE=quincy-3fe4105, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-06-18T02:42:01.978 INFO:journalctl@ceph.osd.5.smithi028.stdout:Jun 18 02:42:01 smithi028 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-5[54541]: 2024-06-18T02:42:01.799+0000 7fe438624640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-06-18T02:41:25.341915+0000 front 2024-06-18T02:41:25.342013+0000 (oldest deadline 2024-06-18T02:41:50.041789+0000) 2024-06-18T02:42:01.978 INFO:journalctl@ceph.osd.5.smithi028.stdout:Jun 18 02:42:01 smithi028 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-5[54541]: 2024-06-18T02:42:01.799+0000 7fe438624640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.1:6814 osd.1 since back 2024-06-18T02:41:36.243895+0000 front 2024-06-18T02:41:36.243836+0000 (oldest deadline 2024-06-18T02:42:00.943734+0000) 2024-06-18T02:42:02.238 INFO:journalctl@ceph.osd.4.smithi028.stdout:Jun 18 02:42:02 smithi028 podman[60170]: 2024-06-18 02:42:01.958198496 +0000 UTC m=+0.018589090 image pull 1a271607156e1b6becab587233eefa647399ff75aa2bb76fdb2f0360ec5b7818 quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032 2024-06-18T02:42:02.238 INFO:journalctl@ceph.osd.4.smithi028.stdout:Jun 18 02:42:02 smithi028 podman[60170]: 2024-06-18 02:42:02.067218553 +0000 UTC m=+0.127609176 container create 40eaed97374a45ca75e0b573dd0a9fed5690f2d23841d0dfbaf0a7f3f0419c49 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240612, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=quincy-3fe4105, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.36.0) 2024-06-18T02:42:02.544 INFO:journalctl@ceph.osd.4.smithi028.stdout:Jun 18 02:42:02 smithi028 podman[60170]: 2024-06-18 02:42:02.277786282 +0000 UTC m=+0.338176881 container init 40eaed97374a45ca75e0b573dd0a9fed5690f2d23841d0dfbaf0a7f3f0419c49 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-4-deactivate, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, RELEASE=quincy-3fe4105, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240612, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-06-18T02:42:02.544 INFO:journalctl@ceph.osd.4.smithi028.stdout:Jun 18 02:42:02 smithi028 podman[60170]: 2024-06-18 02:42:02.281070564 +0000 UTC m=+0.341461149 container start 40eaed97374a45ca75e0b573dd0a9fed5690f2d23841d0dfbaf0a7f3f0419c49 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-4-deactivate, org.label-schema.vendor=CentOS, RELEASE=quincy-3fe4105, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240612, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD) 2024-06-18T02:42:02.545 INFO:journalctl@ceph.osd.4.smithi028.stdout:Jun 18 02:42:02 smithi028 podman[60170]: 2024-06-18 02:42:02.377507328 +0000 UTC m=+0.437897925 container attach 40eaed97374a45ca75e0b573dd0a9fed5690f2d23841d0dfbaf0a7f3f0419c49 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-4-deactivate, io.buildah.version=1.36.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240612, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=quincy-3fe4105, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-06-18T02:42:02.846 INFO:journalctl@ceph.osd.4.smithi028.stdout:Jun 18 02:42:02 smithi028 podman[60170]: 2024-06-18 02:42:02.545636692 +0000 UTC m=+0.606027285 container died 40eaed97374a45ca75e0b573dd0a9fed5690f2d23841d0dfbaf0a7f3f0419c49 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-4-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=quincy-3fe4105, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_CLEAN=True, org.label-schema.build-date=20240612, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0) 2024-06-18T02:42:02.846 INFO:journalctl@ceph.osd.5.smithi028.stdout:Jun 18 02:42:02 smithi028 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-5[54541]: 2024-06-18T02:42:02.786+0000 7fe438624640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-06-18T02:41:25.341915+0000 front 2024-06-18T02:41:25.342013+0000 (oldest deadline 2024-06-18T02:41:50.041789+0000) 2024-06-18T02:42:02.847 INFO:journalctl@ceph.osd.5.smithi028.stdout:Jun 18 02:42:02 smithi028 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-5[54541]: 2024-06-18T02:42:02.786+0000 7fe438624640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.1:6814 osd.1 since back 2024-06-18T02:41:36.243895+0000 front 2024-06-18T02:41:36.243836+0000 (oldest deadline 2024-06-18T02:42:00.943734+0000) 2024-06-18T02:42:03.128 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968@osd.4.service' 2024-06-18T02:42:03.173 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-18T02:42:03.173 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-06-18T02:42:03.173 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-06-18T02:42:03.173 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968@osd.5 2024-06-18T02:42:03.488 INFO:journalctl@ceph.osd.5.smithi028.stdout:Jun 18 02:42:03 smithi028 systemd[1]: Stopping Ceph osd.5 for 7eefb63e-2d1a-11ef-bc9d-c7b262605968... 2024-06-18T02:42:03.760 INFO:journalctl@ceph.osd.5.smithi028.stdout:Jun 18 02:42:03 smithi028 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-5[54541]: 2024-06-18T02:42:03.492+0000 7fe43a809640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-18T02:42:03.761 INFO:journalctl@ceph.osd.5.smithi028.stdout:Jun 18 02:42:03 smithi028 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-5[54541]: 2024-06-18T02:42:03.492+0000 7fe43a809640 -1 osd.5 40 *** Got signal Terminated *** 2024-06-18T02:42:03.761 INFO:journalctl@ceph.osd.5.smithi028.stdout:Jun 18 02:42:03 smithi028 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-5[54541]: 2024-06-18T02:42:03.492+0000 7fe43a809640 -1 osd.5 40 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-18T02:42:03.761 INFO:journalctl@ceph.osd.5.smithi028.stdout:Jun 18 02:42:03 smithi028 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-5[54541]: 2024-06-18T02:42:03.760+0000 7fe438624640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-06-18T02:41:25.341915+0000 front 2024-06-18T02:41:25.342013+0000 (oldest deadline 2024-06-18T02:41:50.041789+0000) 2024-06-18T02:42:03.761 INFO:journalctl@ceph.osd.5.smithi028.stdout:Jun 18 02:42:03 smithi028 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-5[54541]: 2024-06-18T02:42:03.760+0000 7fe438624640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.1:6814 osd.1 since back 2024-06-18T02:41:36.243895+0000 front 2024-06-18T02:41:36.243836+0000 (oldest deadline 2024-06-18T02:42:00.943734+0000) 2024-06-18T02:42:04.987 INFO:journalctl@ceph.osd.5.smithi028.stdout:Jun 18 02:42:04 smithi028 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-5[54541]: 2024-06-18T02:42:04.718+0000 7fe438624640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-06-18T02:41:25.341915+0000 front 2024-06-18T02:41:25.342013+0000 (oldest deadline 2024-06-18T02:41:50.041789+0000) 2024-06-18T02:42:04.988 INFO:journalctl@ceph.osd.5.smithi028.stdout:Jun 18 02:42:04 smithi028 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-5[54541]: 2024-06-18T02:42:04.718+0000 7fe438624640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.1:6814 osd.1 since back 2024-06-18T02:41:36.243895+0000 front 2024-06-18T02:41:36.243836+0000 (oldest deadline 2024-06-18T02:42:00.943734+0000) 2024-06-18T02:42:05.988 INFO:journalctl@ceph.osd.5.smithi028.stdout:Jun 18 02:42:05 smithi028 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-5[54541]: 2024-06-18T02:42:05.691+0000 7fe438624640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-06-18T02:41:25.341915+0000 front 2024-06-18T02:41:25.342013+0000 (oldest deadline 2024-06-18T02:41:50.041789+0000) 2024-06-18T02:42:05.988 INFO:journalctl@ceph.osd.5.smithi028.stdout:Jun 18 02:42:05 smithi028 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-5[54541]: 2024-06-18T02:42:05.691+0000 7fe438624640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.1:6814 osd.1 since back 2024-06-18T02:41:36.243895+0000 front 2024-06-18T02:41:36.243836+0000 (oldest deadline 2024-06-18T02:42:00.943734+0000) 2024-06-18T02:42:05.988 INFO:journalctl@ceph.osd.5.smithi028.stdout:Jun 18 02:42:05 smithi028 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-5[54541]: 2024-06-18T02:42:05.691+0000 7fe438624640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.1:6822 osd.2 since back 2024-06-18T02:41:40.944207+0000 front 2024-06-18T02:41:40.944169+0000 (oldest deadline 2024-06-18T02:42:05.044226+0000) 2024-06-18T02:42:06.988 INFO:journalctl@ceph.osd.5.smithi028.stdout:Jun 18 02:42:06 smithi028 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-5[54541]: 2024-06-18T02:42:06.652+0000 7fe438624640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-06-18T02:41:25.341915+0000 front 2024-06-18T02:41:25.342013+0000 (oldest deadline 2024-06-18T02:41:50.041789+0000) 2024-06-18T02:42:06.988 INFO:journalctl@ceph.osd.5.smithi028.stdout:Jun 18 02:42:06 smithi028 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-5[54541]: 2024-06-18T02:42:06.652+0000 7fe438624640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.1:6814 osd.1 since back 2024-06-18T02:41:36.243895+0000 front 2024-06-18T02:41:36.243836+0000 (oldest deadline 2024-06-18T02:42:00.943734+0000) 2024-06-18T02:42:06.988 INFO:journalctl@ceph.osd.5.smithi028.stdout:Jun 18 02:42:06 smithi028 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-5[54541]: 2024-06-18T02:42:06.652+0000 7fe438624640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.1:6822 osd.2 since back 2024-06-18T02:41:40.944207+0000 front 2024-06-18T02:41:40.944169+0000 (oldest deadline 2024-06-18T02:42:05.044226+0000) 2024-06-18T02:42:07.988 INFO:journalctl@ceph.osd.5.smithi028.stdout:Jun 18 02:42:07 smithi028 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-5[54541]: 2024-06-18T02:42:07.669+0000 7fe438624640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.1:6806 osd.0 since back 2024-06-18T02:41:25.341915+0000 front 2024-06-18T02:41:25.342013+0000 (oldest deadline 2024-06-18T02:41:50.041789+0000) 2024-06-18T02:42:07.988 INFO:journalctl@ceph.osd.5.smithi028.stdout:Jun 18 02:42:07 smithi028 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-5[54541]: 2024-06-18T02:42:07.669+0000 7fe438624640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.1:6814 osd.1 since back 2024-06-18T02:41:36.243895+0000 front 2024-06-18T02:41:36.243836+0000 (oldest deadline 2024-06-18T02:42:00.943734+0000) 2024-06-18T02:42:07.988 INFO:journalctl@ceph.osd.5.smithi028.stdout:Jun 18 02:42:07 smithi028 ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-5[54541]: 2024-06-18T02:42:07.669+0000 7fe438624640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.1:6822 osd.2 since back 2024-06-18T02:41:40.944207+0000 front 2024-06-18T02:41:40.944169+0000 (oldest deadline 2024-06-18T02:42:05.044226+0000) 2024-06-18T02:42:09.216 INFO:journalctl@ceph.osd.5.smithi028.stdout:Jun 18 02:42:08 smithi028 podman[60274]: 2024-06-18 02:42:08.925754939 +0000 UTC m=+5.559948422 container died 4b67db080c5bc4d275b3785bdb32b49a6455b6699a1389715a903215212f76a0 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240612, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, RELEASE=quincy-3fe4105, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0) 2024-06-18T02:42:09.216 INFO:journalctl@ceph.osd.5.smithi028.stdout:Jun 18 02:42:09 smithi028 podman[60274]: 2024-06-18 02:42:09.217255926 +0000 UTC m=+5.851449408 container cleanup 4b67db080c5bc4d275b3785bdb32b49a6455b6699a1389715a903215212f76a0 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-5, org.label-schema.vendor=CentOS, RELEASE=quincy-3fe4105, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240612, GIT_BRANCH=HEAD) 2024-06-18T02:42:09.488 INFO:journalctl@ceph.osd.5.smithi028.stdout:Jun 18 02:42:09 smithi028 bash[60274]: ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-5 2024-06-18T02:42:09.488 INFO:journalctl@ceph.osd.5.smithi028.stdout:Jun 18 02:42:09 smithi028 podman[60292]: 2024-06-18 02:42:09.384901204 +0000 UTC m=+0.454353202 container remove 4b67db080c5bc4d275b3785bdb32b49a6455b6699a1389715a903215212f76a0 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-5, io.buildah.version=1.36.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240612, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-3fe4105) 2024-06-18T02:42:10.129 INFO:journalctl@ceph.osd.5.smithi028.stdout:Jun 18 02:42:09 smithi028 podman[60380]: 2024-06-18 02:42:09.835658777 +0000 UTC m=+0.107649785 container create 65061a5af4f06ea6224c514837fb7c1e5469fc896ce0ec9220b86b7211e797a0 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-5-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=quincy-3fe4105, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240612, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-06-18T02:42:10.129 INFO:journalctl@ceph.osd.5.smithi028.stdout:Jun 18 02:42:09 smithi028 podman[60380]: 2024-06-18 02:42:09.745866117 +0000 UTC m=+0.017857127 image pull 1a271607156e1b6becab587233eefa647399ff75aa2bb76fdb2f0360ec5b7818 quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032 2024-06-18T02:42:10.129 INFO:journalctl@ceph.osd.5.smithi028.stdout:Jun 18 02:42:10 smithi028 podman[60380]: 2024-06-18 02:42:10.054420814 +0000 UTC m=+0.326411841 container init 65061a5af4f06ea6224c514837fb7c1e5469fc896ce0ec9220b86b7211e797a0 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-5-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, RELEASE=quincy-3fe4105, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.build-date=20240612) 2024-06-18T02:42:10.129 INFO:journalctl@ceph.osd.5.smithi028.stdout:Jun 18 02:42:10 smithi028 podman[60380]: 2024-06-18 02:42:10.057722505 +0000 UTC m=+0.329713521 container start 65061a5af4f06ea6224c514837fb7c1e5469fc896ce0ec9220b86b7211e797a0 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, RELEASE=quincy-3fe4105, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240612, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.license=GPLv2) 2024-06-18T02:42:10.129 INFO:journalctl@ceph.osd.5.smithi028.stdout:Jun 18 02:42:10 smithi028 podman[60380]: 2024-06-18 02:42:10.129437447 +0000 UTC m=+0.401428473 container attach 65061a5af4f06ea6224c514837fb7c1e5469fc896ce0ec9220b86b7211e797a0 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-5-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240612, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, RELEASE=quincy-3fe4105, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-06-18T02:42:10.488 INFO:journalctl@ceph.osd.5.smithi028.stdout:Jun 18 02:42:10 smithi028 podman[60380]: 2024-06-18 02:42:10.313665252 +0000 UTC m=+0.585656264 container died 65061a5af4f06ea6224c514837fb7c1e5469fc896ce0ec9220b86b7211e797a0 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968-osd-5-deactivate, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240612, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=quincy-3fe4105, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-18T02:42:10.803 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-7eefb63e-2d1a-11ef-bc9d-c7b262605968@osd.5.service' 2024-06-18T02:42:10.842 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-18T02:42:10.842 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-06-18T02:42:10.842 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 --force --keep-logs 2024-06-18T02:42:20.239 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 --force --keep-logs 2024-06-18T02:42:25.525 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-18T02:42:25.560 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-18T02:42:25.588 INFO:tasks.cephadm:Archiving crash dumps... 2024-06-18T02:42:25.590 DEBUG:teuthology.misc:Transferring archived files from smithi001:/var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-06-17_21:08:03-orch-squid-distro-default-smithi/7760435/remote/smithi001/crash 2024-06-18T02:42:25.591 DEBUG:teuthology.orchestra.run.smithi001:> sudo tar c -f - -C /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/crash -- . 2024-06-18T02:42:25.697 INFO:teuthology.orchestra.run.smithi001.stderr:tar: /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/crash: Cannot open: No such file or directory 2024-06-18T02:42:25.697 INFO:teuthology.orchestra.run.smithi001.stderr:tar: Error is not recoverable: exiting now 2024-06-18T02:42:25.699 DEBUG:teuthology.misc:Transferring archived files from smithi028:/var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-06-17_21:08:03-orch-squid-distro-default-smithi/7760435/remote/smithi028/crash 2024-06-18T02:42:25.700 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/crash -- . 2024-06-18T02:42:25.748 INFO:teuthology.orchestra.run.smithi028.stderr:tar: /var/lib/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/crash: Cannot open: No such file or directory 2024-06-18T02:42:25.748 INFO:teuthology.orchestra.run.smithi028.stderr:tar: Error is not recoverable: exiting now 2024-06-18T02:42:25.749 INFO:tasks.cephadm:Checking cluster log for badness... 2024-06-18T02:42:25.750 DEBUG:teuthology.orchestra.run.smithi001:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v OSD_DOWN | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | head -n 1 2024-06-18T02:42:25.790 INFO:teuthology.orchestra.run.smithi001.stdout:1718677879.2662814 mon.smithi001 (mon.0) 116 : cluster [WRN] Health check failed: 3 mgr modules have recently crashed (RECENT_MGR_MODULE_CRASH) 2024-06-18T02:42:25.791 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-06-18T02:42:25.791 DEBUG:teuthology.orchestra.run.smithi001:> sudo egrep '\[SEC\]' /var/log/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v OSD_DOWN | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | head -n 1 2024-06-18T02:42:25.869 DEBUG:teuthology.orchestra.run.smithi001:> sudo egrep '\[ERR\]' /var/log/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v OSD_DOWN | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | head -n 1 2024-06-18T02:42:25.949 DEBUG:teuthology.orchestra.run.smithi001:> sudo egrep '\[WRN\]' /var/log/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v OSD_DOWN | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | head -n 1 2024-06-18T02:42:26.026 INFO:teuthology.orchestra.run.smithi001.stdout:1718677879.2662814 mon.smithi001 (mon.0) 116 : cluster [WRN] Health check failed: 3 mgr modules have recently crashed (RECENT_MGR_MODULE_CRASH) 2024-06-18T02:42:26.027 INFO:tasks.cephadm:Compressing logs... 2024-06-18T02:42:26.028 DEBUG:teuthology.orchestra.run.smithi001:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-06-18T02:42:26.070 DEBUG:teuthology.orchestra.run.smithi028:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-06-18T02:42:26.094 INFO:teuthology.orchestra.run.smithi001.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/ceph.log 2024-06-18T02:42:26.094 INFO:teuthology.orchestra.run.smithi001.stderr:: No such file or directory 2024-06-18T02:42:26.094 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/ceph-osd.2.log 2024-06-18T02:42:26.094 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/ceph.audit.log 2024-06-18T02:42:26.094 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/ceph-mon.smithi001.log 2024-06-18T02:42:26.097 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/ceph-osd.2.log: /var/log/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/ceph-osd.1.log 2024-06-18T02:42:26.097 INFO:teuthology.orchestra.run.smithi001.stderr: 87.5% -- replaced with /var/log/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/ceph.log.gz 2024-06-18T02:42:26.097 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/ceph-mon.smithi001.log: gzip -5 --verbose -- /var/log/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/ceph-osd.0.log 2024-06-18T02:42:26.097 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/ceph-mgr.smithi001.teoawz.log 2024-06-18T02:42:26.097 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/ceph.cephadm.log 2024-06-18T02:42:26.097 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/ceph-mgr.smithi001.teoawz.log: gzip -5 --verbose -- /var/log/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/ceph-volume.log 2024-06-18T02:42:26.097 INFO:teuthology.orchestra.run.smithi001.stderr: 90.0% -- replaced with /var/log/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/ceph.audit.log.gz 2024-06-18T02:42:26.097 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/ceph.cephadm.log: 83.4% -- replaced with /var/log/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/ceph.cephadm.log.gz 2024-06-18T02:42:26.098 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-06-18T02:42:26.099 INFO:teuthology.orchestra.run.smithi028.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/ceph.log 2024-06-18T02:42:26.099 INFO:teuthology.orchestra.run.smithi028.stderr:: No such file or directory 2024-06-18T02:42:26.099 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/ceph.audit.log 2024-06-18T02:42:26.099 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/ceph-osd.5.log 2024-06-18T02:42:26.099 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/ceph-osd.3.log 2024-06-18T02:42:26.099 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/ceph-osd.4.log 2024-06-18T02:42:26.101 INFO:teuthology.orchestra.run.smithi028.stderr: 87.8% -- replaced with /var/log/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/ceph.log.gz 2024-06-18T02:42:26.101 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/ceph.cephadm.log 2024-06-18T02:42:26.101 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/ceph-osd.4.log: 90.2% -- replaced with /var/log/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/ceph.audit.log.gz 2024-06-18T02:42:26.102 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/ceph-mon.smithi028.log 2024-06-18T02:42:26.102 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/ceph-volume.log 2024-06-18T02:42:26.102 INFO:teuthology.orchestra.run.smithi028.stderr: 81.6% -- replaced with /var/log/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/ceph.cephadm.log.gz 2024-06-18T02:42:26.102 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/ceph-mon.smithi028.log: gzip -5 --verbose -- /var/log/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/ceph-mgr.smithi028.xnuopx.log 2024-06-18T02:42:26.102 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-06-18T02:42:26.102 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/ceph-mgr.smithi028.xnuopx.log: /var/log/ceph/cephadm.log: 92.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-06-18T02:42:26.106 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/ceph-volume.log: /var/log/ceph/cephadm.log: 93.6% -- replaced with /var/log/ceph/cephadm.log.gz 2024-06-18T02:42:26.107 INFO:teuthology.orchestra.run.smithi028.stderr: 92.4% -- replaced with /var/log/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/ceph-mgr.smithi028.xnuopx.log.gz 2024-06-18T02:42:26.107 INFO:teuthology.orchestra.run.smithi001.stderr: 90.9% -- replaced with /var/log/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/ceph-volume.log.gz 2024-06-18T02:42:26.111 INFO:teuthology.orchestra.run.smithi028.stderr: 90.8% -- replaced with /var/log/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/ceph-volume.log.gz 2024-06-18T02:42:26.144 INFO:teuthology.orchestra.run.smithi028.stderr: 92.9% -- replaced with /var/log/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/ceph-osd.5.log.gz 2024-06-18T02:42:26.148 INFO:teuthology.orchestra.run.smithi028.stderr: 92.9% -- replaced with /var/log/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/ceph-osd.4.log.gz 2024-06-18T02:42:26.155 INFO:teuthology.orchestra.run.smithi001.stderr: 93.0% -- replaced with /var/log/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/ceph-osd.2.log.gz 2024-06-18T02:42:26.158 INFO:teuthology.orchestra.run.smithi028.stderr: 92.5% -- replaced with /var/log/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/ceph-osd.3.log.gz 2024-06-18T02:42:26.185 INFO:teuthology.orchestra.run.smithi028.stderr: 92.8% -- replaced with /var/log/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/ceph-mon.smithi028.log.gz 2024-06-18T02:42:26.186 INFO:teuthology.orchestra.run.smithi001.stderr: 92.7% -- replaced with /var/log/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/ceph-osd.0.log.gz 2024-06-18T02:42:26.187 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-06-18T02:42:26.187 INFO:teuthology.orchestra.run.smithi028.stderr:real 0m0.100s 2024-06-18T02:42:26.187 INFO:teuthology.orchestra.run.smithi028.stderr:user 0m0.262s 2024-06-18T02:42:26.187 INFO:teuthology.orchestra.run.smithi028.stderr:sys 0m0.028s 2024-06-18T02:42:26.213 INFO:teuthology.orchestra.run.smithi001.stderr: 93.2% -- replaced with /var/log/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/ceph-osd.1.log.gz 2024-06-18T02:42:26.220 INFO:teuthology.orchestra.run.smithi001.stderr: 89.3% -- replaced with /var/log/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/ceph-mgr.smithi001.teoawz.log.gz 2024-06-18T02:42:26.373 INFO:teuthology.orchestra.run.smithi001.stderr: 90.5% -- replaced with /var/log/ceph/7eefb63e-2d1a-11ef-bc9d-c7b262605968/ceph-mon.smithi001.log.gz 2024-06-18T02:42:26.375 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-06-18T02:42:26.375 INFO:teuthology.orchestra.run.smithi001.stderr:real 0m0.293s 2024-06-18T02:42:26.375 INFO:teuthology.orchestra.run.smithi001.stderr:user 0m0.683s 2024-06-18T02:42:26.375 INFO:teuthology.orchestra.run.smithi001.stderr:sys 0m0.047s 2024-06-18T02:42:26.376 INFO:tasks.cephadm:Archiving logs... 2024-06-18T02:42:26.376 DEBUG:teuthology.misc:Transferring archived files from smithi001:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-06-17_21:08:03-orch-squid-distro-default-smithi/7760435/remote/smithi001/log 2024-06-18T02:42:26.377 DEBUG:teuthology.orchestra.run.smithi001:> sudo tar c -f - -C /var/log/ceph -- . 2024-06-18T02:42:26.546 DEBUG:teuthology.misc:Transferring archived files from smithi028:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-06-17_21:08:03-orch-squid-distro-default-smithi/7760435/remote/smithi028/log 2024-06-18T02:42:26.547 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /var/log/ceph -- . 2024-06-18T02:42:26.621 INFO:tasks.cephadm:Removing cluster... 2024-06-18T02:42:26.621 DEBUG:teuthology.orchestra.run.smithi001:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 --force 2024-06-18T02:42:27.126 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 7eefb63e-2d1a-11ef-bc9d-c7b262605968 --force 2024-06-18T02:42:27.632 INFO:tasks.cephadm:Removing cephadm ... 2024-06-18T02:42:27.632 DEBUG:teuthology.orchestra.run.smithi001:> rm -rf /home/ubuntu/cephtest/cephadm 2024-06-18T02:42:27.651 DEBUG:teuthology.orchestra.run.smithi028:> rm -rf /home/ubuntu/cephtest/cephadm 2024-06-18T02:42:27.667 INFO:tasks.cephadm:Teardown complete 2024-06-18T02:42:27.668 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8e9714173de9e92c97e8ef1045d333e96b793454/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_f18f3f1cbd11e4a41c557e616d172f24deb91473/qa/tasks/cephadm.py", line 2237, in task healthy(ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_ceph_f18f3f1cbd11e4a41c557e616d172f24deb91473/qa/tasks/ceph.py", line 1478, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/git.ceph.com_ceph_f18f3f1cbd11e4a41c557e616d172f24deb91473/qa/tasks/ceph_manager.py", line 3204, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2024-06-18T02:42:27.841 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=965e5ad4623741a193b461a69672b719 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8e9714173de9e92c97e8ef1045d333e96b793454/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_f18f3f1cbd11e4a41c557e616d172f24deb91473/qa/tasks/cephadm.py", line 2237, in task healthy(ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_ceph_f18f3f1cbd11e4a41c557e616d172f24deb91473/qa/tasks/ceph.py", line 1478, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/git.ceph.com_ceph_f18f3f1cbd11e4a41c557e616d172f24deb91473/qa/tasks/ceph_manager.py", line 3204, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2024-06-18T02:42:27.844 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-06-18T02:42:27.855 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-06-18T02:42:27.878 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-06-18T02:42:27.878 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-06-18T02:42:27.890 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-06-18T02:42:27.964 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-06-18T02:42:27.973 INFO:teuthology.task.clock:Checking final clock skew... 2024-06-18T02:42:27.973 DEBUG:teuthology.orchestra.run.smithi001:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-18T02:42:27.976 DEBUG:teuthology.orchestra.run.smithi028:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-18T02:42:27.991 INFO:teuthology.orchestra.run.smithi028.stderr:bash: line 1: ntpq: command not found 2024-06-18T02:42:27.991 INFO:teuthology.orchestra.run.smithi001.stderr:bash: line 1: ntpq: command not found 2024-06-18T02:42:27.995 INFO:teuthology.orchestra.run.smithi028.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-18T02:42:27.995 INFO:teuthology.orchestra.run.smithi028.stdout:=============================================================================== 2024-06-18T02:42:27.996 INFO:teuthology.orchestra.run.smithi028.stdout:^+ hv01.front.sepia.ceph.com 3 6 377 51 +535us[ +535us] +/- 69ms 2024-06-18T02:42:27.996 INFO:teuthology.orchestra.run.smithi028.stdout:^* hv02.front.sepia.ceph.com 3 6 377 52 -37us[ -26us] +/- 56ms 2024-06-18T02:42:27.996 INFO:teuthology.orchestra.run.smithi028.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 49 -519us[ -519us] +/- 72ms 2024-06-18T02:42:27.996 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-06-18T02:42:27.997 INFO:teuthology.orchestra.run.smithi001.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-18T02:42:27.998 INFO:teuthology.orchestra.run.smithi001.stdout:=============================================================================== 2024-06-18T02:42:27.998 INFO:teuthology.orchestra.run.smithi001.stdout:^+ hv01.front.sepia.ceph.com 3 6 377 52 +550us[ +576us] +/- 69ms 2024-06-18T02:42:27.998 INFO:teuthology.orchestra.run.smithi001.stdout:^* hv02.front.sepia.ceph.com 3 6 377 49 -30us[-4490ns] +/- 56ms 2024-06-18T02:42:27.998 INFO:teuthology.orchestra.run.smithi001.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 52 -511us[ -486us] +/- 72ms 2024-06-18T02:42:27.998 INFO:teuthology.orchestra.run.smithi001.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-06-18T02:42:27.998 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-06-18T02:42:28.032 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-06-18T02:42:28.032 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-06-18T02:42:28.041 DEBUG:teuthology.orchestra.run.smithi001:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-06-18T02:42:28.121 DEBUG:teuthology.orchestra.run.smithi028:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-06-18T02:42:28.190 DEBUG:teuthology.orchestra.run.smithi001:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-18T02:42:28.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-18T02:42:28.222 DEBUG:teuthology.orchestra.run.smithi028:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-18T02:42:28.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-18T02:42:28.251 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-06-18T02:42:28.260 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-06-18T02:42:28.268 INFO:teuthology.task.internal:Duration was 1237.093862 seconds 2024-06-18T02:42:28.269 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-06-18T02:42:28.277 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-06-18T02:42:28.277 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-06-18T02:42:28.280 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-06-18T02:42:28.321 INFO:teuthology.orchestra.run.smithi001.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-18T02:42:28.331 INFO:teuthology.orchestra.run.smithi028.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-18T02:42:28.771 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-06-18T02:42:28.772 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi001.front.sepia.ceph.com 2024-06-18T02:42:28.772 DEBUG:teuthology.orchestra.run.smithi001:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-06-18T02:42:28.798 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi028.front.sepia.ceph.com 2024-06-18T02:42:28.799 DEBUG:teuthology.orchestra.run.smithi028:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-06-18T02:42:28.836 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-06-18T02:42:28.836 DEBUG:teuthology.orchestra.run.smithi001:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-06-18T02:42:28.841 DEBUG:teuthology.orchestra.run.smithi028:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-06-18T02:42:28.905 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-06-18T02:42:28.905 DEBUG:teuthology.orchestra.run.smithi001:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-06-18T02:42:28.914 DEBUG:teuthology.orchestra.run.smithi028:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-06-18T02:42:29.306 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-06-18T02:42:29.316 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-06-18T02:42:29.317 DEBUG:teuthology.orchestra.run.smithi001:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-06-18T02:42:29.342 DEBUG:teuthology.orchestra.run.smithi028:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-06-18T02:42:29.369 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-06-18T02:42:29.378 DEBUG:teuthology.orchestra.run.smithi001:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-06-18T02:42:29.385 DEBUG:teuthology.orchestra.run.smithi028:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-06-18T02:42:29.406 INFO:teuthology.orchestra.run.smithi001.stdout:kernel.core_pattern = core 2024-06-18T02:42:29.433 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern = core 2024-06-18T02:42:29.453 DEBUG:teuthology.orchestra.run.smithi001:> test -e /home/ubuntu/cephtest/archive/coredump 2024-06-18T02:42:29.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-18T02:42:29.485 DEBUG:teuthology.orchestra.run.smithi028:> test -e /home/ubuntu/cephtest/archive/coredump 2024-06-18T02:42:29.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-18T02:42:29.509 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-06-18T02:42:29.518 INFO:teuthology.task.internal:Transferring archived files... 2024-06-18T02:42:29.519 DEBUG:teuthology.misc:Transferring archived files from smithi001:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-06-17_21:08:03-orch-squid-distro-default-smithi/7760435/remote/smithi001 2024-06-18T02:42:29.519 DEBUG:teuthology.orchestra.run.smithi001:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-06-18T02:42:29.565 DEBUG:teuthology.misc:Transferring archived files from smithi028:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-06-17_21:08:03-orch-squid-distro-default-smithi/7760435/remote/smithi028 2024-06-18T02:42:29.565 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-06-18T02:42:29.601 INFO:teuthology.task.internal:Removing archive directory... 2024-06-18T02:42:29.601 DEBUG:teuthology.orchestra.run.smithi001:> rm -rf -- /home/ubuntu/cephtest/archive 2024-06-18T02:42:29.603 DEBUG:teuthology.orchestra.run.smithi028:> rm -rf -- /home/ubuntu/cephtest/archive 2024-06-18T02:42:29.650 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-06-18T02:42:29.660 INFO:teuthology.task.internal:Not uploading archives. 2024-06-18T02:42:29.660 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-06-18T02:42:29.669 INFO:teuthology.task.internal:Tidying up after the test... 2024-06-18T02:42:29.669 DEBUG:teuthology.orchestra.run.smithi001:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-06-18T02:42:29.672 DEBUG:teuthology.orchestra.run.smithi028:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-06-18T02:42:29.685 INFO:teuthology.orchestra.run.smithi001.stdout: 262165 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 18 02:42 /home/ubuntu/cephtest 2024-06-18T02:42:29.706 INFO:teuthology.orchestra.run.smithi028.stdout: 265303 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 18 02:42 /home/ubuntu/cephtest 2024-06-18T02:42:29.707 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-06-18T02:42:29.717 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-06-18T02:42:29.740 INFO:teuthology.run:Summary data: description: orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} duration: 1237.0938622951508 failure_reason: '"1718677879.2662814 mon.smithi001 (mon.0) 116 : cluster [WRN] Health check failed: 3 mgr modules have recently crashed (RECENT_MGR_MODULE_CRASH)" in cluster log' flavor: default owner: scheduled_teuthology@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=965e5ad4623741a193b461a69672b719 status: fail success: false 2024-06-18T02:42:29.740 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-06-18T02:42:29.824 INFO:teuthology.run:FAIL