2024-06-18T02:40:37.256 INFO:root:teuthology version: 0.0.1.dev325+g8e97141 2024-06-18T02:40:37.256 DEBUG:teuthology.run:Teuthology command: teuthology --description orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} --verbose --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2024-06-17_21:08:03-orch-squid-distro-default-smithi/7760463 --name teuthology-2024-06-17_21:08:03-orch-squid-distro-default-smithi -- /home/teuthworker/archive/teuthology-2024-06-17_21:08:03-orch-squid-distro-default-smithi/7760463/orig.config.yaml 2024-06-18T02:40:37.311 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-06-18T02:40:37.407 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-06-17_21:08:03-orch-squid-distro-default-smithi/7760463 branch: squid description: orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7760463' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/quincy ' name: teuthology-2024-06-17_21:08:03-orch-squid-distro-default-smithi no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - OSD_DOWN - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - Degraded data redundancy sha1: 3781e3a592387d16e8020f5db99cdee4198f6d00 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 3781e3a592387d16e8020f5db99cdee4198f6d00 kclient: syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: f18f3f1cbd11e4a41c557e616d172f24deb91473 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 4712 sha1: 3781e3a592387d16e8020f5db99cdee4198f6d00 sleep_before_teardown: 0 subset: 61/64 suite: orch suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_f18f3f1cbd11e4a41c557e616d172f24deb91473/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: f18f3f1cbd11e4a41c557e616d172f24deb91473 targets: smithi016.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMNam4f/IWYmRVQijB1Kzp5xr/qLA3XdRnzu7yF1CTAIlQFUvCRjwIyT/hbuq4oLgM7vthCoRSpPrSLRW/N3vVc= smithi123.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBtULjn7wx7vk5WlUfOXM6KgiYm/8AVVryW7HGIv9ski2LNJJCvVPfN8UdKvOMzmqwUCmgdk3Jv+cq9dH2K/ibc= tasks: - install: branch: quincy exclude_packages: - ceph-volume - print: '**** done install task...' - cephadm: cephadm_branch: quincy cephadm_git_url: https://github.com/ceph/ceph conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.ceph.io/ceph-ci/ceph:quincy roleless: true - print: '**** done end installing quincy cephadm ...' - cephadm.shell: host.a: - ceph config set mgr mgr/cephadm/use_repo_digest true --force - print: '**** done cephadm.shell ceph config set mgr...' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 1 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay false - cephadm.shell: host.a: - ceph fs set cephfs inline_data true --yes-i-really-really-mean-it - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - ceph-fuse: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] variables: fail_fs: true teuthology_branch: main teuthology_sha1: 8e9714173de9e92c97e8ef1045d333e96b793454 timestamp: 2024-06-17_21:08:03 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph versions | jq -e '.mgr | length == 1' - ceph versions | jq -e '.mgr | keys' | grep $sha1 - ceph versions | jq -e '.overall | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 2' - ceph orch ps - cephadm.shell: env: - sha1 host.a: - ceph config set mgr mgr/orchestrator/fail_fs true - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2698841 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-06-18T02:40:37.407 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_f18f3f1cbd11e4a41c557e616d172f24deb91473/qa; will attempt to use it 2024-06-18T02:40:37.408 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_f18f3f1cbd11e4a41c557e616d172f24deb91473/qa/tasks 2024-06-18T02:40:37.408 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-06-18T02:40:37.410 INFO:teuthology.task.internal:Checking packages... 2024-06-18T02:40:37.432 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '3781e3a592387d16e8020f5db99cdee4198f6d00' 2024-06-18T02:40:37.432 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-18T02:40:37.433 INFO:teuthology.packaging:ref: None 2024-06-18T02:40:37.433 INFO:teuthology.packaging:tag: None 2024-06-18T02:40:37.433 INFO:teuthology.packaging:branch: squid 2024-06-18T02:40:37.433 INFO:teuthology.packaging:sha1: 3781e3a592387d16e8020f5db99cdee4198f6d00 2024-06-18T02:40:37.433 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-06-18T02:40:37.621 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3145.gf18f3f1c 2024-06-18T02:40:37.623 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-06-18T02:40:37.629 INFO:teuthology.task.internal:no buildpackages task found 2024-06-18T02:40:37.629 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-06-18T02:40:37.646 INFO:teuthology.task.internal:Saving configuration 2024-06-18T02:40:37.666 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-06-18T02:40:37.673 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-06-18T02:40:37.698 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi016.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-06-17_21:08:03-orch-squid-distro-default-smithi/7760463', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-06-18 02:35:51.664877', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMNam4f/IWYmRVQijB1Kzp5xr/qLA3XdRnzu7yF1CTAIlQFUvCRjwIyT/hbuq4oLgM7vthCoRSpPrSLRW/N3vVc='} 2024-06-18T02:40:37.719 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi123.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-06-17_21:08:03-orch-squid-distro-default-smithi/7760463', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-06-18 02:35:51.662960', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBtULjn7wx7vk5WlUfOXM6KgiYm/8AVVryW7HGIv9ski2LNJJCvVPfN8UdKvOMzmqwUCmgdk3Jv+cq9dH2K/ibc='} 2024-06-18T02:40:37.719 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-06-18T02:40:37.725 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-06-18T02:40:37.726 INFO:teuthology.task.internal:roles: ubuntu@smithi123.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-06-18T02:40:37.726 INFO:teuthology.run_tasks:Running task console_log... 2024-06-18T02:40:37.789 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f26053b7ac0>, signals=[15]) 2024-06-18T02:40:37.789 INFO:teuthology.run_tasks:Running task internal.connect... 2024-06-18T02:40:37.795 INFO:teuthology.task.internal:Opening connections... 2024-06-18T02:40:37.796 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi016.front.sepia.ceph.com 2024-06-18T02:40:37.797 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-18T02:40:37.874 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi123.front.sepia.ceph.com 2024-06-18T02:40:37.875 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-18T02:40:37.948 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-06-18T02:40:37.955 DEBUG:teuthology.orchestra.run.smithi016:> uname -m 2024-06-18T02:40:37.973 INFO:teuthology.orchestra.run.smithi016.stdout:x86_64 2024-06-18T02:40:37.973 DEBUG:teuthology.orchestra.run.smithi016:> cat /etc/os-release 2024-06-18T02:40:38.031 INFO:teuthology.orchestra.run.smithi016.stdout:NAME="CentOS Stream" 2024-06-18T02:40:38.032 INFO:teuthology.orchestra.run.smithi016.stdout:VERSION="9" 2024-06-18T02:40:38.032 INFO:teuthology.orchestra.run.smithi016.stdout:ID="centos" 2024-06-18T02:40:38.032 INFO:teuthology.orchestra.run.smithi016.stdout:ID_LIKE="rhel fedora" 2024-06-18T02:40:38.032 INFO:teuthology.orchestra.run.smithi016.stdout:VERSION_ID="9" 2024-06-18T02:40:38.032 INFO:teuthology.orchestra.run.smithi016.stdout:PLATFORM_ID="platform:el9" 2024-06-18T02:40:38.032 INFO:teuthology.orchestra.run.smithi016.stdout:PRETTY_NAME="CentOS Stream 9" 2024-06-18T02:40:38.032 INFO:teuthology.orchestra.run.smithi016.stdout:ANSI_COLOR="0;31" 2024-06-18T02:40:38.032 INFO:teuthology.orchestra.run.smithi016.stdout:LOGO="fedora-logo-icon" 2024-06-18T02:40:38.032 INFO:teuthology.orchestra.run.smithi016.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-06-18T02:40:38.032 INFO:teuthology.orchestra.run.smithi016.stdout:HOME_URL="https://centos.org/" 2024-06-18T02:40:38.032 INFO:teuthology.orchestra.run.smithi016.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-06-18T02:40:38.032 INFO:teuthology.orchestra.run.smithi016.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-06-18T02:40:38.032 INFO:teuthology.orchestra.run.smithi016.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-06-18T02:40:38.033 INFO:teuthology.lock.ops:Updating smithi016.front.sepia.ceph.com on lock server 2024-06-18T02:40:38.059 DEBUG:teuthology.orchestra.run.smithi123:> uname -m 2024-06-18T02:40:38.077 INFO:teuthology.orchestra.run.smithi123.stdout:x86_64 2024-06-18T02:40:38.077 DEBUG:teuthology.orchestra.run.smithi123:> cat /etc/os-release 2024-06-18T02:40:38.134 INFO:teuthology.orchestra.run.smithi123.stdout:NAME="CentOS Stream" 2024-06-18T02:40:38.134 INFO:teuthology.orchestra.run.smithi123.stdout:VERSION="9" 2024-06-18T02:40:38.134 INFO:teuthology.orchestra.run.smithi123.stdout:ID="centos" 2024-06-18T02:40:38.135 INFO:teuthology.orchestra.run.smithi123.stdout:ID_LIKE="rhel fedora" 2024-06-18T02:40:38.135 INFO:teuthology.orchestra.run.smithi123.stdout:VERSION_ID="9" 2024-06-18T02:40:38.135 INFO:teuthology.orchestra.run.smithi123.stdout:PLATFORM_ID="platform:el9" 2024-06-18T02:40:38.135 INFO:teuthology.orchestra.run.smithi123.stdout:PRETTY_NAME="CentOS Stream 9" 2024-06-18T02:40:38.135 INFO:teuthology.orchestra.run.smithi123.stdout:ANSI_COLOR="0;31" 2024-06-18T02:40:38.135 INFO:teuthology.orchestra.run.smithi123.stdout:LOGO="fedora-logo-icon" 2024-06-18T02:40:38.135 INFO:teuthology.orchestra.run.smithi123.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-06-18T02:40:38.135 INFO:teuthology.orchestra.run.smithi123.stdout:HOME_URL="https://centos.org/" 2024-06-18T02:40:38.135 INFO:teuthology.orchestra.run.smithi123.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-06-18T02:40:38.135 INFO:teuthology.orchestra.run.smithi123.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-06-18T02:40:38.135 INFO:teuthology.orchestra.run.smithi123.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-06-18T02:40:38.136 INFO:teuthology.lock.ops:Updating smithi123.front.sepia.ceph.com on lock server 2024-06-18T02:40:38.159 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-06-18T02:40:38.168 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-06-18T02:40:38.173 INFO:teuthology.task.internal:Checking for old test directory... 2024-06-18T02:40:38.174 DEBUG:teuthology.orchestra.run.smithi016:> test '!' -e /home/ubuntu/cephtest 2024-06-18T02:40:38.176 DEBUG:teuthology.orchestra.run.smithi123:> test '!' -e /home/ubuntu/cephtest 2024-06-18T02:40:38.193 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-06-18T02:40:38.199 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-06-18T02:40:38.199 DEBUG:teuthology.orchestra.run.smithi016:> test -z $(ls -A /var/lib/ceph) 2024-06-18T02:40:38.234 DEBUG:teuthology.orchestra.run.smithi123:> test -z $(ls -A /var/lib/ceph) 2024-06-18T02:40:38.332 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-06-18T02:40:38.383 INFO:teuthology.run_tasks:Running task kernel... 2024-06-18T02:40:38.398 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-06-18T02:40:38.398 INFO:teuthology.task.kernel:config {'host.a': {'kdb': 1, 'sha1': 'distro'}, 'host.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-06-18T02:40:38.401 DEBUG:teuthology.orchestra.run.smithi016:> test -f /run/.containerenv -o -f /.dockerenv 2024-06-18T02:40:38.401 DEBUG:teuthology.orchestra.run.smithi123:> test -f /run/.containerenv -o -f /.dockerenv 2024-06-18T02:40:38.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-18T02:40:38.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-18T02:40:38.419 DEBUG:teuthology.orchestra.run.smithi123:> uname -r 2024-06-18T02:40:38.420 DEBUG:teuthology.orchestra.run.smithi016:> uname -r 2024-06-18T02:40:38.476 INFO:teuthology.orchestra.run.smithi123.stdout:5.14.0-437.el9.x86_64 2024-06-18T02:40:38.477 INFO:teuthology.task.kernel:Running kernel on smithi123: 5.14.0-437.el9.x86_64 2024-06-18T02:40:38.477 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum install -y kernel 2024-06-18T02:40:38.478 INFO:teuthology.orchestra.run.smithi016.stdout:5.14.0-437.el9.x86_64 2024-06-18T02:40:38.478 INFO:teuthology.task.kernel:Running kernel on smithi016: 5.14.0-437.el9.x86_64 2024-06-18T02:40:38.478 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum install -y kernel 2024-06-18T02:40:42.593 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 9 - BaseOS 3.2 MB/s | 8.1 MB 00:02 2024-06-18T02:40:44.150 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - BaseOS 2.0 MB/s | 8.1 MB 00:04 2024-06-18T02:40:52.188 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 9 - AppStream 2.3 MB/s | 20 MB 00:08 2024-06-18T02:40:53.406 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - AppStream 2.4 MB/s | 20 MB 00:08 2024-06-18T02:40:57.677 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 9 - CRB 4.0 MB/s | 6.3 MB 00:01 2024-06-18T02:40:59.303 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 9 - Extras packages 62 kB/s | 17 kB 00:00 2024-06-18T02:40:59.509 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - CRB 2.8 MB/s | 6.3 MB 00:02 2024-06-18T02:41:00.448 INFO:teuthology.orchestra.run.smithi123.stdout:Extra Packages for Enterprise Linux 21 MB/s | 21 MB 00:00 2024-06-18T02:41:01.091 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - Extras packages 51 kB/s | 17 kB 00:00 2024-06-18T02:41:01.945 INFO:teuthology.orchestra.run.smithi016.stdout:Extra Packages for Enterprise Linux 30 MB/s | 21 MB 00:00 2024-06-18T02:41:05.754 INFO:teuthology.orchestra.run.smithi123.stdout:lab-extras 17 kB/s | 1.7 kB 00:00 2024-06-18T02:41:07.334 INFO:teuthology.orchestra.run.smithi016.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-06-18T02:41:07.852 INFO:teuthology.orchestra.run.smithi123.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-06-18T02:41:07.852 INFO:teuthology.orchestra.run.smithi123.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-06-18T02:41:07.942 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-06-18T02:41:07.951 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-06-18T02:41:07.951 INFO:teuthology.orchestra.run.smithi123.stdout: Package Architecture Version Repository Size 2024-06-18T02:41:07.952 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-06-18T02:41:07.952 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-06-18T02:41:07.952 INFO:teuthology.orchestra.run.smithi123.stdout: kernel x86_64 5.14.0-460.el9 baseos 1.1 M 2024-06-18T02:41:07.952 INFO:teuthology.orchestra.run.smithi123.stdout:Installing dependencies: 2024-06-18T02:41:07.952 INFO:teuthology.orchestra.run.smithi123.stdout: kernel-core x86_64 5.14.0-460.el9 baseos 16 M 2024-06-18T02:41:07.952 INFO:teuthology.orchestra.run.smithi123.stdout: kernel-modules x86_64 5.14.0-460.el9 baseos 35 M 2024-06-18T02:41:07.952 INFO:teuthology.orchestra.run.smithi123.stdout: kernel-modules-core x86_64 5.14.0-460.el9 baseos 29 M 2024-06-18T02:41:07.952 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:41:07.952 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-06-18T02:41:07.952 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-06-18T02:41:07.952 INFO:teuthology.orchestra.run.smithi123.stdout:Install 4 Packages 2024-06-18T02:41:07.952 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:41:07.953 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 81 M 2024-06-18T02:41:07.953 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 124 M 2024-06-18T02:41:07.953 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-06-18T02:41:08.297 INFO:teuthology.orchestra.run.smithi123.stdout:(1/4): kernel-5.14.0-460.el9.x86_64.rpm 3.4 MB/s | 1.1 MB 00:00 2024-06-18T02:41:09.441 INFO:teuthology.orchestra.run.smithi016.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-06-18T02:41:09.442 INFO:teuthology.orchestra.run.smithi016.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-06-18T02:41:09.539 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-06-18T02:41:09.548 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-06-18T02:41:09.548 INFO:teuthology.orchestra.run.smithi016.stdout: Package Architecture Version Repository Size 2024-06-18T02:41:09.548 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-06-18T02:41:09.548 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-06-18T02:41:09.548 INFO:teuthology.orchestra.run.smithi016.stdout: kernel x86_64 5.14.0-460.el9 baseos 1.1 M 2024-06-18T02:41:09.548 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2024-06-18T02:41:09.548 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-core x86_64 5.14.0-460.el9 baseos 16 M 2024-06-18T02:41:09.549 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-modules x86_64 5.14.0-460.el9 baseos 35 M 2024-06-18T02:41:09.549 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-modules-core x86_64 5.14.0-460.el9 baseos 29 M 2024-06-18T02:41:09.549 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:41:09.549 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-06-18T02:41:09.549 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-06-18T02:41:09.549 INFO:teuthology.orchestra.run.smithi016.stdout:Install 4 Packages 2024-06-18T02:41:09.549 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:41:09.549 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 81 M 2024-06-18T02:41:09.550 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 124 M 2024-06-18T02:41:09.550 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-06-18T02:41:09.880 INFO:teuthology.orchestra.run.smithi016.stdout:[MIRROR] kernel-5.14.0-460.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-460.el9.x86_64.rpm (IP: 216.31.2.234) 2024-06-18T02:41:09.880 INFO:teuthology.orchestra.run.smithi016.stdout:[MIRROR] kernel-core-5.14.0-460.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-460.el9.x86_64.rpm (IP: 216.31.2.234) 2024-06-18T02:41:09.880 INFO:teuthology.orchestra.run.smithi016.stdout:[MIRROR] kernel-modules-5.14.0-460.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-460.el9.x86_64.rpm (IP: 216.31.2.234) 2024-06-18T02:41:10.020 INFO:teuthology.orchestra.run.smithi016.stdout:[MIRROR] kernel-5.14.0-460.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-460.el9.x86_64.rpm (IP: 216.31.2.234) 2024-06-18T02:41:10.022 INFO:teuthology.orchestra.run.smithi016.stdout:[MIRROR] kernel-core-5.14.0-460.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-460.el9.x86_64.rpm (IP: 216.31.2.234) 2024-06-18T02:41:10.048 INFO:teuthology.orchestra.run.smithi016.stdout:[MIRROR] kernel-modules-5.14.0-460.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-460.el9.x86_64.rpm (IP: 216.31.2.234) 2024-06-18T02:41:10.319 INFO:teuthology.orchestra.run.smithi016.stdout:(1/4): kernel-5.14.0-460.el9.x86_64.rpm 2.1 MB/s | 1.1 MB 00:00 2024-06-18T02:41:10.347 INFO:teuthology.orchestra.run.smithi016.stdout:[MIRROR] kernel-modules-core-5.14.0-460.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-460.el9.x86_64.rpm (IP: 216.31.2.234) 2024-06-18T02:41:10.376 INFO:teuthology.orchestra.run.smithi016.stdout:[MIRROR] kernel-modules-core-5.14.0-460.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-460.el9.x86_64.rpm (IP: 216.31.2.234) 2024-06-18T02:41:12.672 INFO:teuthology.orchestra.run.smithi123.stdout:(2/4): kernel-core-5.14.0-460.el9.x86_64.rpm 3.4 MB/s | 16 MB 00:04 2024-06-18T02:41:12.719 INFO:teuthology.orchestra.run.smithi016.stdout:(2/4): kernel-core-5.14.0-460.el9.x86_64.rpm 5.6 MB/s | 16 MB 00:02 2024-06-18T02:41:14.994 INFO:teuthology.orchestra.run.smithi016.stdout:(3/4): kernel-modules-core-5.14.0-460.el9.x86_6 6.2 MB/s | 29 MB 00:04 2024-06-18T02:41:15.469 INFO:teuthology.orchestra.run.smithi016.stdout:(4/4): kernel-modules-5.14.0-460.el9.x86_64.rpm 6.2 MB/s | 35 MB 00:05 2024-06-18T02:41:15.472 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-06-18T02:41:15.473 INFO:teuthology.orchestra.run.smithi016.stdout:Total 14 MB/s | 81 MB 00:05 2024-06-18T02:41:16.212 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-06-18T02:41:16.639 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-06-18T02:41:16.639 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-06-18T02:41:16.871 INFO:teuthology.orchestra.run.smithi123.stdout:(3/4): kernel-modules-core-5.14.0-460.el9.x86_6 3.4 MB/s | 29 MB 00:08 2024-06-18T02:41:17.653 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-06-18T02:41:17.653 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-06-18T02:41:18.038 INFO:teuthology.orchestra.run.smithi123.stdout:(4/4): kernel-modules-5.14.0-460.el9.x86_64.rpm 3.5 MB/s | 35 MB 00:10 2024-06-18T02:41:18.039 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-06-18T02:41:18.039 INFO:teuthology.orchestra.run.smithi123.stdout:Total 8.0 MB/s | 81 MB 00:10 2024-06-18T02:41:18.747 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-06-18T02:41:19.080 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-06-18T02:41:19.080 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-06-18T02:41:20.064 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-06-18T02:41:20.064 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-06-18T02:41:20.210 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-06-18T02:41:20.840 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : kernel-modules-core-5.14.0-460.el9.x86_64 1/4 2024-06-18T02:41:20.920 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : kernel-core-5.14.0-460.el9.x86_64 2/4 2024-06-18T02:41:22.470 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-06-18T02:41:23.158 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : kernel-modules-core-5.14.0-460.el9.x86_64 1/4 2024-06-18T02:41:23.255 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : kernel-core-5.14.0-460.el9.x86_64 2/4 2024-06-18T02:41:23.745 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: kernel-core-5.14.0-460.el9.x86_64 2/4 2024-06-18T02:41:23.890 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : kernel-modules-5.14.0-460.el9.x86_64 3/4 2024-06-18T02:41:25.941 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: kernel-core-5.14.0-460.el9.x86_64 2/4 2024-06-18T02:41:26.081 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : kernel-modules-5.14.0-460.el9.x86_64 3/4 2024-06-18T02:41:29.005 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: kernel-modules-5.14.0-460.el9.x86_64 3/4 2024-06-18T02:41:29.064 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : kernel-5.14.0-460.el9.x86_64 4/4 2024-06-18T02:41:30.823 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: kernel-modules-5.14.0-460.el9.x86_64 3/4 2024-06-18T02:41:30.870 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : kernel-5.14.0-460.el9.x86_64 4/4 2024-06-18T02:41:34.184 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: kernel-modules-core-5.14.0-460.el9.x86_64 4/4 2024-06-18T02:41:35.614 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: kernel-modules-core-5.14.0-460.el9.x86_64 4/4 2024-06-18T02:42:05.066 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: kernel-core-5.14.0-460.el9.x86_64 4/4 2024-06-18T02:42:05.106 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: kernel-modules-5.14.0-460.el9.x86_64 4/4 2024-06-18T02:42:05.318 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: kernel-core-5.14.0-460.el9.x86_64 4/4 2024-06-18T02:42:05.360 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: kernel-modules-5.14.0-460.el9.x86_64 4/4 2024-06-18T02:42:05.483 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: kernel-5.14.0-460.el9.x86_64 4/4 2024-06-18T02:42:05.483 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : kernel-5.14.0-460.el9.x86_64 1/4 2024-06-18T02:42:05.483 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : kernel-core-5.14.0-460.el9.x86_64 2/4 2024-06-18T02:42:05.483 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : kernel-modules-5.14.0-460.el9.x86_64 3/4 2024-06-18T02:42:05.718 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: kernel-5.14.0-460.el9.x86_64 4/4 2024-06-18T02:42:05.718 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : kernel-5.14.0-460.el9.x86_64 1/4 2024-06-18T02:42:05.718 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : kernel-core-5.14.0-460.el9.x86_64 2/4 2024-06-18T02:42:05.718 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : kernel-modules-5.14.0-460.el9.x86_64 3/4 2024-06-18T02:42:06.278 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : kernel-modules-core-5.14.0-460.el9.x86_64 4/4 2024-06-18T02:42:06.278 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:42:06.278 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-06-18T02:42:06.278 INFO:teuthology.orchestra.run.smithi123.stdout: kernel-5.14.0-460.el9.x86_64 2024-06-18T02:42:06.278 INFO:teuthology.orchestra.run.smithi123.stdout: kernel-core-5.14.0-460.el9.x86_64 2024-06-18T02:42:06.278 INFO:teuthology.orchestra.run.smithi123.stdout: kernel-modules-5.14.0-460.el9.x86_64 2024-06-18T02:42:06.279 INFO:teuthology.orchestra.run.smithi123.stdout: kernel-modules-core-5.14.0-460.el9.x86_64 2024-06-18T02:42:06.279 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:42:06.279 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-06-18T02:42:06.612 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : kernel-modules-core-5.14.0-460.el9.x86_64 4/4 2024-06-18T02:42:06.612 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:42:06.612 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-06-18T02:42:06.612 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-5.14.0-460.el9.x86_64 2024-06-18T02:42:06.612 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-core-5.14.0-460.el9.x86_64 2024-06-18T02:42:06.612 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-modules-5.14.0-460.el9.x86_64 2024-06-18T02:42:06.613 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-modules-core-5.14.0-460.el9.x86_64 2024-06-18T02:42:06.613 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:42:06.613 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-06-18T02:42:06.673 DEBUG:teuthology.orchestra.run.smithi123:> rpm -q kernel | sort -rV | head -n 1 2024-06-18T02:42:06.748 INFO:teuthology.orchestra.run.smithi123.stdout:kernel-5.14.0-460.el9.x86_64 2024-06-18T02:42:06.748 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-460.el9.x86_64 2024-06-18T02:42:06.748 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-460.el9.x86_64 2024-06-18T02:42:06.748 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-06-18T02:42:06.748 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-06-18T02:42:06.749 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-06-18T02:42:06.749 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi123.front.sepia.ceph.com, path=None, version=distro) 2024-06-18T02:42:06.749 DEBUG:teuthology.orchestra.run.smithi123:> rpm -q kernel | sort -rV | head -n 1 2024-06-18T02:42:06.803 INFO:teuthology.orchestra.run.smithi123.stdout:kernel-5.14.0-460.el9.x86_64 2024-06-18T02:42:06.804 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-460.el9.x86_64 2024-06-18T02:42:06.804 DEBUG:teuthology.orchestra.run.smithi123:> sudo rpm -qi grub2-tools 2024-06-18T02:42:06.836 INFO:teuthology.orchestra.run.smithi123.stdout:Name : grub2-tools 2024-06-18T02:42:06.837 INFO:teuthology.orchestra.run.smithi123.stdout:Epoch : 1 2024-06-18T02:42:06.837 INFO:teuthology.orchestra.run.smithi123.stdout:Version : 2.06 2024-06-18T02:42:06.837 INFO:teuthology.orchestra.run.smithi123.stdout:Release : 68.el9 2024-06-18T02:42:06.837 INFO:teuthology.orchestra.run.smithi123.stdout:Architecture: x86_64 2024-06-18T02:42:06.837 INFO:teuthology.orchestra.run.smithi123.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-06-18T02:42:06.837 INFO:teuthology.orchestra.run.smithi123.stdout:Group : Unspecified 2024-06-18T02:42:06.837 INFO:teuthology.orchestra.run.smithi123.stdout:Size : 8264254 2024-06-18T02:42:06.837 INFO:teuthology.orchestra.run.smithi123.stdout:License : GPLv3+ 2024-06-18T02:42:06.837 INFO:teuthology.orchestra.run.smithi123.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-06-18T02:42:06.837 INFO:teuthology.orchestra.run.smithi123.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-06-18T02:42:06.837 INFO:teuthology.orchestra.run.smithi123.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-06-18T02:42:06.837 INFO:teuthology.orchestra.run.smithi123.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-06-18T02:42:06.837 INFO:teuthology.orchestra.run.smithi123.stdout:Packager : builder@centos.org 2024-06-18T02:42:06.838 INFO:teuthology.orchestra.run.smithi123.stdout:Vendor : CentOS 2024-06-18T02:42:06.838 INFO:teuthology.orchestra.run.smithi123.stdout:URL : http://www.gnu.org/software/grub/ 2024-06-18T02:42:06.838 INFO:teuthology.orchestra.run.smithi123.stdout:Summary : Support tools for GRUB. 2024-06-18T02:42:06.838 INFO:teuthology.orchestra.run.smithi123.stdout:Description : 2024-06-18T02:42:06.838 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:42:06.838 INFO:teuthology.orchestra.run.smithi123.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-06-18T02:42:06.838 INFO:teuthology.orchestra.run.smithi123.stdout:customizable bootloader with modular architecture. It supports a rich 2024-06-18T02:42:06.838 INFO:teuthology.orchestra.run.smithi123.stdout:variety of kernel formats, file systems, computer architectures and 2024-06-18T02:42:06.838 INFO:teuthology.orchestra.run.smithi123.stdout:hardware devices. 2024-06-18T02:42:06.838 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:42:06.838 INFO:teuthology.orchestra.run.smithi123.stdout:This subpackage provides tools for support of all platforms. 2024-06-18T02:42:06.839 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-06-18T02:42:06.839 INFO:teuthology.task.kernel:Updating grub on smithi123 to boot 5.14.0-460.el9.x86_64 2024-06-18T02:42:06.839 DEBUG:teuthology.orchestra.run.smithi123:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-06-18T02:42:06.991 DEBUG:teuthology.orchestra.run.smithi016:> rpm -q kernel | sort -rV | head -n 1 2024-06-18T02:42:07.066 INFO:teuthology.orchestra.run.smithi016.stdout:kernel-5.14.0-460.el9.x86_64 2024-06-18T02:42:07.067 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-460.el9.x86_64 2024-06-18T02:42:07.067 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-460.el9.x86_64 2024-06-18T02:42:07.067 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-06-18T02:42:07.067 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-06-18T02:42:07.068 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-06-18T02:42:07.068 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi016.front.sepia.ceph.com, path=None, version=distro) 2024-06-18T02:42:07.068 DEBUG:teuthology.orchestra.run.smithi016:> rpm -q kernel | sort -rV | head -n 1 2024-06-18T02:42:07.123 INFO:teuthology.orchestra.run.smithi016.stdout:kernel-5.14.0-460.el9.x86_64 2024-06-18T02:42:07.124 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-460.el9.x86_64 2024-06-18T02:42:07.124 DEBUG:teuthology.orchestra.run.smithi016:> sudo rpm -qi grub2-tools 2024-06-18T02:42:07.155 INFO:teuthology.orchestra.run.smithi016.stdout:Name : grub2-tools 2024-06-18T02:42:07.155 INFO:teuthology.orchestra.run.smithi016.stdout:Epoch : 1 2024-06-18T02:42:07.155 INFO:teuthology.orchestra.run.smithi016.stdout:Version : 2.06 2024-06-18T02:42:07.155 INFO:teuthology.orchestra.run.smithi016.stdout:Release : 68.el9 2024-06-18T02:42:07.156 INFO:teuthology.orchestra.run.smithi016.stdout:Architecture: x86_64 2024-06-18T02:42:07.156 INFO:teuthology.orchestra.run.smithi016.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-06-18T02:42:07.156 INFO:teuthology.orchestra.run.smithi016.stdout:Group : Unspecified 2024-06-18T02:42:07.156 INFO:teuthology.orchestra.run.smithi016.stdout:Size : 8264254 2024-06-18T02:42:07.156 INFO:teuthology.orchestra.run.smithi016.stdout:License : GPLv3+ 2024-06-18T02:42:07.156 INFO:teuthology.orchestra.run.smithi016.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-06-18T02:42:07.156 INFO:teuthology.orchestra.run.smithi016.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-06-18T02:42:07.156 INFO:teuthology.orchestra.run.smithi016.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-06-18T02:42:07.156 INFO:teuthology.orchestra.run.smithi016.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-06-18T02:42:07.156 INFO:teuthology.orchestra.run.smithi016.stdout:Packager : builder@centos.org 2024-06-18T02:42:07.156 INFO:teuthology.orchestra.run.smithi016.stdout:Vendor : CentOS 2024-06-18T02:42:07.156 INFO:teuthology.orchestra.run.smithi016.stdout:URL : http://www.gnu.org/software/grub/ 2024-06-18T02:42:07.156 INFO:teuthology.orchestra.run.smithi016.stdout:Summary : Support tools for GRUB. 2024-06-18T02:42:07.157 INFO:teuthology.orchestra.run.smithi016.stdout:Description : 2024-06-18T02:42:07.157 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:42:07.157 INFO:teuthology.orchestra.run.smithi016.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-06-18T02:42:07.157 INFO:teuthology.orchestra.run.smithi016.stdout:customizable bootloader with modular architecture. It supports a rich 2024-06-18T02:42:07.157 INFO:teuthology.orchestra.run.smithi016.stdout:variety of kernel formats, file systems, computer architectures and 2024-06-18T02:42:07.157 INFO:teuthology.orchestra.run.smithi016.stdout:hardware devices. 2024-06-18T02:42:07.157 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:42:07.157 INFO:teuthology.orchestra.run.smithi016.stdout:This subpackage provides tools for support of all platforms. 2024-06-18T02:42:07.158 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-06-18T02:42:07.158 INFO:teuthology.task.kernel:Updating grub on smithi016 to boot 5.14.0-460.el9.x86_64 2024-06-18T02:42:07.158 DEBUG:teuthology.orchestra.run.smithi016:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-06-18T02:42:07.444 INFO:teuthology.orchestra.run.smithi123.stderr:Generating grub configuration file ... 2024-06-18T02:42:07.961 INFO:teuthology.orchestra.run.smithi016.stderr:Generating grub configuration file ... 2024-06-18T02:42:08.848 INFO:teuthology.orchestra.run.smithi123.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-06-18T02:42:08.878 INFO:teuthology.orchestra.run.smithi123.stderr:done 2024-06-18T02:42:08.880 DEBUG:teuthology.orchestra.run.smithi123:> mktemp 2024-06-18T02:42:08.895 INFO:teuthology.orchestra.run.smithi123.stdout:/tmp/tmp.SI8yIIxSDK 2024-06-18T02:42:08.895 DEBUG:teuthology.orchestra.run.smithi123:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.SI8yIIxSDK 2024-06-18T02:42:08.961 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 0666 /tmp/tmp.SI8yIIxSDK 2024-06-18T02:42:09.094 DEBUG:teuthology.orchestra.remote:smithi123:/tmp/tmp.SI8yIIxSDK is 6KB 2024-06-18T02:42:09.142 DEBUG:teuthology.orchestra.run.smithi123:> rm -fr /tmp/tmp.SI8yIIxSDK 2024-06-18T02:42:09.157 DEBUG:teuthology.orchestra.run.smithi123:> sudo /bin/ls /boot/loader/entries || true 2024-06-18T02:42:09.218 INFO:teuthology.orchestra.run.smithi123.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-06-18T02:42:09.218 INFO:teuthology.orchestra.run.smithi123.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-06-18T02:42:09.218 INFO:teuthology.orchestra.run.smithi123.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-06-18T02:42:09.218 INFO:teuthology.orchestra.run.smithi123.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-460.el9.x86_64.conf 2024-06-18T02:42:09.220 DEBUG:teuthology.orchestra.run.smithi123:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-460.el9.x86_64 2024-06-18T02:42:09.369 DEBUG:teuthology.orchestra.run.smithi123:> sudo shutdown -r now 2024-06-18T02:42:09.384 INFO:teuthology.orchestra.run.smithi016.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-06-18T02:42:09.415 INFO:teuthology.orchestra.run.smithi016.stderr:done 2024-06-18T02:42:09.418 DEBUG:teuthology.orchestra.run.smithi016:> mktemp 2024-06-18T02:42:09.433 INFO:teuthology.orchestra.run.smithi016.stdout:/tmp/tmp.TS2qVNhmVW 2024-06-18T02:42:09.433 DEBUG:teuthology.orchestra.run.smithi016:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.TS2qVNhmVW 2024-06-18T02:42:09.497 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 0666 /tmp/tmp.TS2qVNhmVW 2024-06-18T02:42:09.635 DEBUG:teuthology.orchestra.remote:smithi016:/tmp/tmp.TS2qVNhmVW is 6KB 2024-06-18T02:42:09.684 DEBUG:teuthology.orchestra.run.smithi016:> rm -fr /tmp/tmp.TS2qVNhmVW 2024-06-18T02:42:09.698 DEBUG:teuthology.orchestra.run.smithi016:> sudo /bin/ls /boot/loader/entries || true 2024-06-18T02:42:09.760 INFO:teuthology.orchestra.run.smithi016.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-06-18T02:42:09.760 INFO:teuthology.orchestra.run.smithi016.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-06-18T02:42:09.760 INFO:teuthology.orchestra.run.smithi016.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-06-18T02:42:09.760 INFO:teuthology.orchestra.run.smithi016.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-460.el9.x86_64.conf 2024-06-18T02:42:09.762 DEBUG:teuthology.orchestra.run.smithi016:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-460.el9.x86_64 2024-06-18T02:42:09.939 DEBUG:teuthology.orchestra.run.smithi016:> sudo shutdown -r now 2024-06-18T02:42:39.398 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-06-18T02:42:39.399 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com' 2024-06-18T02:42:39.399 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-18T02:42:39.942 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-06-18T02:42:39.943 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-06-18T02:42:39.943 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-18T02:42:57.951 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.123 2024-06-18T02:42:58.462 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.16 2024-06-18T02:43:06.956 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com' 2024-06-18T02:43:06.957 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-18T02:43:07.463 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-06-18T02:43:07.464 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-18T02:44:07.019 DEBUG:teuthology.orchestra.remote:timed out 2024-06-18T02:44:07.466 DEBUG:teuthology.orchestra.remote:timed out 2024-06-18T02:44:19.030 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com' 2024-06-18T02:44:19.031 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-18T02:44:19.422 DEBUG:teuthology.orchestra.run.smithi123:> true 2024-06-18T02:44:19.467 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-06-18T02:44:19.468 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-18T02:44:19.802 DEBUG:teuthology.orchestra.run.smithi016:> true 2024-06-18T02:44:19.832 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi123.front.sepia.ceph.com' 2024-06-18T02:44:19.833 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.14.0-460.el9.x86_64"... 2024-06-18T02:44:19.833 DEBUG:teuthology.orchestra.run.smithi123:> uname -r 2024-06-18T02:44:19.847 INFO:teuthology.orchestra.run.smithi123.stdout:5.14.0-460.el9.x86_64 2024-06-18T02:44:19.847 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-460.el9.x86_64 vs 5.14.0-460.el9.x86_64 2024-06-18T02:44:19.848 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-06-18T02:44:19.848 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-06-18T02:44:20.152 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-06-18T02:44:20.153 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.14.0-460.el9.x86_64"... 2024-06-18T02:44:20.153 DEBUG:teuthology.orchestra.run.smithi016:> uname -r 2024-06-18T02:44:20.211 INFO:teuthology.orchestra.run.smithi016.stdout:5.14.0-460.el9.x86_64 2024-06-18T02:44:20.211 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-460.el9.x86_64 vs 5.14.0-460.el9.x86_64 2024-06-18T02:44:20.211 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-06-18T02:44:20.211 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-06-18T02:44:20.848 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-06-18T02:44:20.848 DEBUG:teuthology.orchestra.run.smithi123:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-06-18T02:44:20.937 INFO:teuthology.orchestra.run.smithi123.stdout:ttyS1 2024-06-18T02:44:20.990 DEBUG:teuthology.parallel:result is None 2024-06-18T02:44:21.212 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-06-18T02:44:21.212 DEBUG:teuthology.orchestra.run.smithi016:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-06-18T02:44:21.276 INFO:teuthology.orchestra.run.smithi016.stdout:ttyS1 2024-06-18T02:44:21.311 DEBUG:teuthology.parallel:result is None 2024-06-18T02:44:21.311 INFO:teuthology.run_tasks:Running task internal.base... 2024-06-18T02:44:21.320 INFO:teuthology.task.internal:Creating test directory... 2024-06-18T02:44:21.320 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-06-18T02:44:21.323 DEBUG:teuthology.orchestra.run.smithi123:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-06-18T02:44:21.339 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-06-18T02:44:21.351 INFO:teuthology.run_tasks:Running task internal.archive... 2024-06-18T02:44:21.360 INFO:teuthology.task.internal:Creating archive directory... 2024-06-18T02:44:21.361 DEBUG:teuthology.orchestra.run.smithi016:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-06-18T02:44:21.382 DEBUG:teuthology.orchestra.run.smithi123:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-06-18T02:44:21.416 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-06-18T02:44:21.423 INFO:teuthology.task.internal:Enabling coredump saving... 2024-06-18T02:44:21.424 DEBUG:teuthology.orchestra.run.smithi016:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-06-18T02:44:21.453 DEBUG:teuthology.orchestra.run.smithi123:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-06-18T02:44:21.486 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-18T02:44:21.498 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-18T02:44:21.507 INFO:teuthology.orchestra.run.smithi123.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-18T02:44:21.517 INFO:teuthology.orchestra.run.smithi123.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-18T02:44:21.519 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-06-18T02:44:21.526 INFO:teuthology.task.internal:Configuring sudo... 2024-06-18T02:44:21.527 DEBUG:teuthology.orchestra.run.smithi016:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-06-18T02:44:21.543 DEBUG:teuthology.orchestra.run.smithi123:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-06-18T02:44:21.569 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-06-18T02:44:21.580 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-06-18T02:44:21.580 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-06-18T02:44:21.612 DEBUG:teuthology.orchestra.run.smithi123:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-06-18T02:44:21.627 DEBUG:teuthology.orchestra.run.smithi016:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-18T02:44:21.714 DEBUG:teuthology.orchestra.run.smithi016:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-18T02:44:21.760 DEBUG:teuthology.orchestra.run.smithi016:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-18T02:44:21.846 DEBUG:teuthology.orchestra.run.smithi016:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-18T02:44:21.886 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-06-18T02:44:21.886 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-06-18T02:44:21.952 DEBUG:teuthology.orchestra.run.smithi123:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-18T02:44:21.992 DEBUG:teuthology.orchestra.run.smithi123:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-18T02:44:22.033 DEBUG:teuthology.orchestra.run.smithi123:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-18T02:44:22.111 DEBUG:teuthology.orchestra.run.smithi123:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-18T02:44:22.148 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-06-18T02:44:22.148 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-06-18T02:44:22.211 DEBUG:teuthology.orchestra.run.smithi016:> sudo service rsyslog restart 2024-06-18T02:44:22.213 DEBUG:teuthology.orchestra.run.smithi123:> sudo service rsyslog restart 2024-06-18T02:44:22.249 INFO:teuthology.orchestra.run.smithi016.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-18T02:44:22.295 INFO:teuthology.orchestra.run.smithi123.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-18T02:44:22.593 INFO:teuthology.run_tasks:Running task internal.timer... 2024-06-18T02:44:22.601 INFO:teuthology.task.internal:Starting timer... 2024-06-18T02:44:22.601 INFO:teuthology.run_tasks:Running task pcp... 2024-06-18T02:44:22.620 INFO:teuthology.run_tasks:Running task selinux... 2024-06-18T02:44:22.630 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-06-18T02:44:22.630 DEBUG:teuthology.orchestra.run.smithi016:> sudo service auditd rotate 2024-06-18T02:44:22.689 INFO:teuthology.orchestra.run.smithi016.stdout:Rotating logs: 2024-06-18T02:44:22.691 DEBUG:teuthology.orchestra.run.smithi123:> sudo service auditd rotate 2024-06-18T02:44:22.769 INFO:teuthology.orchestra.run.smithi123.stdout:Rotating logs: 2024-06-18T02:44:22.770 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-06-18T02:44:22.771 DEBUG:teuthology.orchestra.run.smithi016:> /usr/sbin/getenforce 2024-06-18T02:44:22.812 INFO:teuthology.orchestra.run.smithi016.stdout:Permissive 2024-06-18T02:44:22.813 DEBUG:teuthology.orchestra.run.smithi123:> /usr/sbin/getenforce 2024-06-18T02:44:22.834 INFO:teuthology.orchestra.run.smithi123.stdout:Permissive 2024-06-18T02:44:22.835 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi016.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi123.front.sepia.ceph.com': 'permissive'} 2024-06-18T02:44:22.835 DEBUG:teuthology.orchestra.run.smithi016:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-18T02:44:22.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-18T02:44:22.882 DEBUG:teuthology.orchestra.run.smithi123:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-18T02:44:22.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-18T02:44:22.906 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-06-18T02:44:22.907 DEBUG:teuthology.orchestra.run.smithi016:> sudo /usr/sbin/setenforce permissive 2024-06-18T02:44:22.957 DEBUG:teuthology.orchestra.run.smithi123:> sudo /usr/sbin/setenforce permissive 2024-06-18T02:44:23.046 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-06-18T02:44:23.055 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-06-18T02:44:23.058 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-06-18T02:44:23.167 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-06-18T02:44:23.178 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-06-18T02:44:23.178 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi016.front.sepia.ceph.com,smithi123.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-06-18T02:48:50.195 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi016.front.sepia.ceph.com'), Remote(name='ubuntu@smithi123.front.sepia.ceph.com')] 2024-06-18T02:48:50.197 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-06-18T02:48:50.198 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-18T02:48:50.273 DEBUG:teuthology.orchestra.run.smithi016:> true 2024-06-18T02:48:50.354 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-06-18T02:48:50.354 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com' 2024-06-18T02:48:50.355 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-18T02:48:50.429 DEBUG:teuthology.orchestra.run.smithi123:> true 2024-06-18T02:48:50.514 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi123.front.sepia.ceph.com' 2024-06-18T02:48:50.515 INFO:teuthology.run_tasks:Running task clock... 2024-06-18T02:48:50.524 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-06-18T02:48:50.524 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-06-18T02:48:50.525 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-18T02:48:50.527 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-06-18T02:48:50.528 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-18T02:48:50.557 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-06-18T02:48:50.573 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-06-18T02:48:50.585 INFO:teuthology.orchestra.run.smithi123.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-06-18T02:48:50.599 INFO:teuthology.orchestra.run.smithi123.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-06-18T02:48:50.600 INFO:teuthology.orchestra.run.smithi016.stderr:sudo: ntpd: command not found 2024-06-18T02:48:50.611 INFO:teuthology.orchestra.run.smithi016.stdout:506 Cannot talk to daemon 2024-06-18T02:48:50.623 INFO:teuthology.orchestra.run.smithi123.stderr:sudo: ntpd: command not found 2024-06-18T02:48:50.627 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-06-18T02:48:50.634 INFO:teuthology.orchestra.run.smithi123.stdout:506 Cannot talk to daemon 2024-06-18T02:48:50.642 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-06-18T02:48:50.648 INFO:teuthology.orchestra.run.smithi123.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-06-18T02:48:50.662 INFO:teuthology.orchestra.run.smithi123.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-06-18T02:48:50.706 INFO:teuthology.orchestra.run.smithi016.stderr:bash: line 1: ntpq: command not found 2024-06-18T02:48:50.709 INFO:teuthology.orchestra.run.smithi016.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-18T02:48:50.709 INFO:teuthology.orchestra.run.smithi016.stdout:=============================================================================== 2024-06-18T02:48:50.709 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-18T02:48:50.709 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-18T02:48:50.709 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-18T02:48:50.709 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-18T02:48:50.714 INFO:teuthology.orchestra.run.smithi123.stderr:bash: line 1: ntpq: command not found 2024-06-18T02:48:50.717 INFO:teuthology.orchestra.run.smithi123.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-18T02:48:50.718 INFO:teuthology.orchestra.run.smithi123.stdout:=============================================================================== 2024-06-18T02:48:50.718 INFO:teuthology.orchestra.run.smithi123.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-18T02:48:50.718 INFO:teuthology.orchestra.run.smithi123.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-18T02:48:50.718 INFO:teuthology.orchestra.run.smithi123.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-18T02:48:50.718 INFO:teuthology.orchestra.run.smithi123.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-18T02:48:50.719 INFO:teuthology.run_tasks:Running task install... 2024-06-18T02:48:50.728 DEBUG:teuthology.task.install:project ceph 2024-06-18T02:48:50.728 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '3781e3a592387d16e8020f5db99cdee4198f6d00'}} 2024-06-18T02:48:50.728 DEBUG:teuthology.task.install:config {'branch': 'quincy', 'exclude_packages': ['ceph-volume'], 'flavor': 'default', 'sha1': '3781e3a592387d16e8020f5db99cdee4198f6d00'} 2024-06-18T02:48:50.728 INFO:teuthology.task.install:Using flavor: default 2024-06-18T02:48:50.733 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-06-18T02:48:50.733 INFO:teuthology.task.install:extra packages: [] 2024-06-18T02:48:50.733 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3781e3a592387d16e8020f5db99cdee4198f6d00', 'tag': None, 'wait_for_package': False} 2024-06-18T02:48:50.734 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-18T02:48:50.734 INFO:teuthology.packaging:ref: None 2024-06-18T02:48:50.734 INFO:teuthology.packaging:tag: None 2024-06-18T02:48:50.734 INFO:teuthology.packaging:branch: quincy 2024-06-18T02:48:50.734 INFO:teuthology.packaging:sha1: 3781e3a592387d16e8020f5db99cdee4198f6d00 2024-06-18T02:48:50.734 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-06-18T02:48:50.736 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3781e3a592387d16e8020f5db99cdee4198f6d00', 'tag': None, 'wait_for_package': False} 2024-06-18T02:48:50.736 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-18T02:48:50.736 INFO:teuthology.packaging:ref: None 2024-06-18T02:48:50.736 INFO:teuthology.packaging:tag: None 2024-06-18T02:48:50.736 INFO:teuthology.packaging:branch: quincy 2024-06-18T02:48:50.736 INFO:teuthology.packaging:sha1: 3781e3a592387d16e8020f5db99cdee4198f6d00 2024-06-18T02:48:50.737 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-06-18T02:48:51.065 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/quincy/3fe410552b224c0fbe57dfbac3e18568c0f8fef6/centos/9/flavors/default/ 2024-06-18T02:48:51.066 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1012.g3fe41055 2024-06-18T02:48:51.082 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/quincy/3fe410552b224c0fbe57dfbac3e18568c0f8fef6/centos/9/flavors/default/ 2024-06-18T02:48:51.082 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1012.g3fe41055 2024-06-18T02:48:51.181 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/quincy/3fe410552b224c0fbe57dfbac3e18568c0f8fef6/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/quincy/3fe410552b224c0fbe57dfbac3e18568c0f8fef6/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/quincy/3fe410552b224c0fbe57dfbac3e18568c0f8fef6/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-06-18T02:48:51.181 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-06-18T02:48:51.181 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-06-18T02:48:51.209 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-06-18T02:48:51.209 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-18T02:48:51.209 INFO:teuthology.packaging:ref: None 2024-06-18T02:48:51.210 INFO:teuthology.packaging:tag: None 2024-06-18T02:48:51.210 INFO:teuthology.packaging:branch: quincy 2024-06-18T02:48:51.210 INFO:teuthology.packaging:sha1: 3781e3a592387d16e8020f5db99cdee4198f6d00 2024-06-18T02:48:51.210 DEBUG:teuthology.orchestra.run.smithi016:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-06-18T02:48:51.268 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/quincy/3fe410552b224c0fbe57dfbac3e18568c0f8fef6/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/quincy/3fe410552b224c0fbe57dfbac3e18568c0f8fef6/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/quincy/3fe410552b224c0fbe57dfbac3e18568c0f8fef6/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-06-18T02:48:51.268 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-06-18T02:48:51.268 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-06-18T02:48:51.277 DEBUG:teuthology.orchestra.run.smithi016:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-06-18T02:48:51.295 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-06-18T02:48:51.295 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-18T02:48:51.295 INFO:teuthology.packaging:ref: None 2024-06-18T02:48:51.295 INFO:teuthology.packaging:tag: None 2024-06-18T02:48:51.295 INFO:teuthology.packaging:branch: quincy 2024-06-18T02:48:51.295 INFO:teuthology.packaging:sha1: 3781e3a592387d16e8020f5db99cdee4198f6d00 2024-06-18T02:48:51.295 DEBUG:teuthology.orchestra.run.smithi123:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-06-18T02:48:51.355 DEBUG:teuthology.orchestra.run.smithi016:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-06-18T02:48:51.362 DEBUG:teuthology.orchestra.run.smithi123:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-06-18T02:48:51.420 INFO:teuthology.orchestra.run.smithi016.stdout:check_obsoletes = 1 2024-06-18T02:48:51.422 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum clean all 2024-06-18T02:48:51.437 DEBUG:teuthology.orchestra.run.smithi123:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-06-18T02:48:51.502 INFO:teuthology.orchestra.run.smithi123.stdout:check_obsoletes = 1 2024-06-18T02:48:51.503 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum clean all 2024-06-18T02:48:51.656 INFO:teuthology.orchestra.run.smithi016.stdout:45 files removed 2024-06-18T02:48:51.684 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-radosgw 2024-06-18T02:48:51.737 INFO:teuthology.orchestra.run.smithi123.stdout:45 files removed 2024-06-18T02:48:51.764 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-radosgw 2024-06-18T02:48:52.314 INFO:teuthology.orchestra.run.smithi016.stdout:ceph packages for x86_64 230 kB/s | 77 kB 00:00 2024-06-18T02:48:52.357 INFO:teuthology.orchestra.run.smithi123.stdout:ceph packages for x86_64 245 kB/s | 77 kB 00:00 2024-06-18T02:48:52.725 INFO:teuthology.orchestra.run.smithi123.stdout:ceph noarch packages 34 kB/s | 11 kB 00:00 2024-06-18T02:48:52.747 INFO:teuthology.orchestra.run.smithi016.stdout:ceph noarch packages 28 kB/s | 11 kB 00:00 2024-06-18T02:48:53.100 INFO:teuthology.orchestra.run.smithi123.stdout:ceph source packages 5.4 kB/s | 1.8 kB 00:00 2024-06-18T02:48:53.190 INFO:teuthology.orchestra.run.smithi016.stdout:ceph source packages 4.5 kB/s | 1.8 kB 00:00 2024-06-18T02:48:54.833 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 9 - BaseOS 4.8 MB/s | 8.1 MB 00:01 2024-06-18T02:48:54.917 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - BaseOS 4.8 MB/s | 8.1 MB 00:01 2024-06-18T02:48:57.851 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - AppStream 11 MB/s | 20 MB 00:01 2024-06-18T02:48:59.019 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 9 - AppStream 6.4 MB/s | 20 MB 00:03 2024-06-18T02:49:02.358 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - CRB 9.3 MB/s | 6.3 MB 00:00 2024-06-18T02:49:03.859 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - Extras packages 71 kB/s | 17 kB 00:00 2024-06-18T02:49:04.738 INFO:teuthology.orchestra.run.smithi016.stdout:Extra Packages for Enterprise Linux 29 MB/s | 21 MB 00:00 2024-06-18T02:49:05.110 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 9 - CRB 2.6 MB/s | 6.3 MB 00:02 2024-06-18T02:49:06.919 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 9 - Extras packages 36 kB/s | 17 kB 00:00 2024-06-18T02:49:07.748 INFO:teuthology.orchestra.run.smithi123.stdout:Extra Packages for Enterprise Linux 31 MB/s | 21 MB 00:00 2024-06-18T02:49:10.153 INFO:teuthology.orchestra.run.smithi016.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-06-18T02:49:12.645 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-06-18T02:49:12.661 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-06-18T02:49:12.661 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2024-06-18T02:49:12.661 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-06-18T02:49:12.661 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-06-18T02:49:12.661 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-radosgw x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 14 M 2024-06-18T02:49:12.661 INFO:teuthology.orchestra.run.smithi016.stdout:Upgrading: 2024-06-18T02:49:12.661 INFO:teuthology.orchestra.run.smithi016.stdout: librados2 x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 3.1 M 2024-06-18T02:49:12.661 INFO:teuthology.orchestra.run.smithi016.stdout: librbd1 x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 2.9 M 2024-06-18T02:49:12.661 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2024-06-18T02:49:12.662 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-base x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 5.0 M 2024-06-18T02:49:12.662 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-common x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 22 M 2024-06-18T02:49:12.662 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-selinux x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 25 k 2024-06-18T02:49:12.662 INFO:teuthology.orchestra.run.smithi016.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-06-18T02:49:12.662 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs2 x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 637 k 2024-06-18T02:49:12.662 INFO:teuthology.orchestra.run.smithi016.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-06-18T02:49:12.662 INFO:teuthology.orchestra.run.smithi016.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-06-18T02:49:12.662 INFO:teuthology.orchestra.run.smithi016.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-06-18T02:49:12.662 INFO:teuthology.orchestra.run.smithi016.stdout: libradosstriper1 x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 458 k 2024-06-18T02:49:12.662 INFO:teuthology.orchestra.run.smithi016.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-06-18T02:49:12.662 INFO:teuthology.orchestra.run.smithi016.stdout: librgw2 x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 4.9 M 2024-06-18T02:49:12.662 INFO:teuthology.orchestra.run.smithi016.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-06-18T02:49:12.663 INFO:teuthology.orchestra.run.smithi016.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-06-18T02:49:12.663 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-argparse x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 45 k 2024-06-18T02:49:12.663 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-common x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 118 k 2024-06-18T02:49:12.663 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cephfs x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 156 k 2024-06-18T02:49:12.663 INFO:teuthology.orchestra.run.smithi016.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-06-18T02:49:12.663 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rados x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 320 k 2024-06-18T02:49:12.663 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rbd x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 293 k 2024-06-18T02:49:12.663 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rgw x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 99 k 2024-06-18T02:49:12.663 INFO:teuthology.orchestra.run.smithi016.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-06-18T02:49:12.663 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:49:12.663 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-06-18T02:49:12.663 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-06-18T02:49:12.663 INFO:teuthology.orchestra.run.smithi016.stdout:Install 22 Packages 2024-06-18T02:49:12.664 INFO:teuthology.orchestra.run.smithi016.stdout:Upgrade 2 Packages 2024-06-18T02:49:12.664 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:49:12.664 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 58 M 2024-06-18T02:49:12.664 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-06-18T02:49:13.221 INFO:teuthology.orchestra.run.smithi123.stdout:lab-extras 5.2 kB/s | 1.7 kB 00:00 2024-06-18T02:49:14.312 INFO:teuthology.orchestra.run.smithi016.stdout:(1/24): ceph-radosgw-17.2.7-1012.g3fe41055.el9. 10 MB/s | 14 MB 00:01 2024-06-18T02:49:14.355 INFO:teuthology.orchestra.run.smithi016.stdout:(2/24): ceph-selinux-17.2.7-1012.g3fe41055.el9. 601 kB/s | 25 kB 00:00 2024-06-18T02:49:14.421 INFO:teuthology.orchestra.run.smithi016.stdout:(3/24): ceph-base-17.2.7-1012.g3fe41055.el9.x86 3.4 MB/s | 5.0 MB 00:01 2024-06-18T02:49:14.455 INFO:teuthology.orchestra.run.smithi016.stdout:(4/24): libcephfs2-17.2.7-1012.g3fe41055.el9.x8 6.2 MB/s | 637 kB 00:00 2024-06-18T02:49:14.555 INFO:teuthology.orchestra.run.smithi016.stdout:(5/24): libradosstriper1-17.2.7-1012.g3fe41055. 3.3 MB/s | 458 kB 00:00 2024-06-18T02:49:14.639 INFO:teuthology.orchestra.run.smithi016.stdout:(6/24): python3-ceph-argparse-17.2.7-1012.g3fe4 540 kB/s | 45 kB 00:00 2024-06-18T02:49:14.789 INFO:teuthology.orchestra.run.smithi016.stdout:(7/24): librgw2-17.2.7-1012.g3fe41055.el9.x86_6 15 MB/s | 4.9 MB 00:00 2024-06-18T02:49:14.814 INFO:teuthology.orchestra.run.smithi016.stdout:(8/24): python3-ceph-common-17.2.7-1012.g3fe410 671 kB/s | 118 kB 00:00 2024-06-18T02:49:14.848 INFO:teuthology.orchestra.run.smithi016.stdout:(9/24): python3-cephfs-17.2.7-1012.g3fe41055.el 2.6 MB/s | 156 kB 00:00 2024-06-18T02:49:14.873 INFO:teuthology.orchestra.run.smithi016.stdout:(10/24): python3-rados-17.2.7-1012.g3fe41055.el 5.3 MB/s | 320 kB 00:00 2024-06-18T02:49:14.915 INFO:teuthology.orchestra.run.smithi016.stdout:(11/24): python3-rbd-17.2.7-1012.g3fe41055.el9. 4.3 MB/s | 293 kB 00:00 2024-06-18T02:49:14.940 INFO:teuthology.orchestra.run.smithi016.stdout:(12/24): python3-rgw-17.2.7-1012.g3fe41055.el9. 1.4 MB/s | 99 kB 00:00 2024-06-18T02:49:15.629 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-06-18T02:49:15.640 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-06-18T02:49:15.640 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repository Size 2024-06-18T02:49:15.640 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-06-18T02:49:15.640 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-06-18T02:49:15.640 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-radosgw x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 14 M 2024-06-18T02:49:15.640 INFO:teuthology.orchestra.run.smithi123.stdout:Upgrading: 2024-06-18T02:49:15.640 INFO:teuthology.orchestra.run.smithi123.stdout: librados2 x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 3.1 M 2024-06-18T02:49:15.640 INFO:teuthology.orchestra.run.smithi123.stdout: librbd1 x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 2.9 M 2024-06-18T02:49:15.640 INFO:teuthology.orchestra.run.smithi123.stdout:Installing dependencies: 2024-06-18T02:49:15.640 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-base x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 5.0 M 2024-06-18T02:49:15.641 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-common x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 22 M 2024-06-18T02:49:15.641 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-selinux x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 25 k 2024-06-18T02:49:15.641 INFO:teuthology.orchestra.run.smithi123.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-06-18T02:49:15.641 INFO:teuthology.orchestra.run.smithi123.stdout: libcephfs2 x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 637 k 2024-06-18T02:49:15.641 INFO:teuthology.orchestra.run.smithi123.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-06-18T02:49:15.641 INFO:teuthology.orchestra.run.smithi123.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-06-18T02:49:15.641 INFO:teuthology.orchestra.run.smithi123.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-06-18T02:49:15.641 INFO:teuthology.orchestra.run.smithi123.stdout: libradosstriper1 x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 458 k 2024-06-18T02:49:15.641 INFO:teuthology.orchestra.run.smithi123.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-06-18T02:49:15.641 INFO:teuthology.orchestra.run.smithi123.stdout: librgw2 x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 4.9 M 2024-06-18T02:49:15.641 INFO:teuthology.orchestra.run.smithi123.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-06-18T02:49:15.641 INFO:teuthology.orchestra.run.smithi123.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-06-18T02:49:15.642 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ceph-argparse x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 45 k 2024-06-18T02:49:15.642 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ceph-common x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 118 k 2024-06-18T02:49:15.642 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cephfs x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 156 k 2024-06-18T02:49:15.642 INFO:teuthology.orchestra.run.smithi123.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-06-18T02:49:15.642 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rados x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 320 k 2024-06-18T02:49:15.642 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rbd x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 293 k 2024-06-18T02:49:15.642 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rgw x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 99 k 2024-06-18T02:49:15.642 INFO:teuthology.orchestra.run.smithi123.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-06-18T02:49:15.642 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:49:15.642 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-06-18T02:49:15.642 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-06-18T02:49:15.642 INFO:teuthology.orchestra.run.smithi123.stdout:Install 22 Packages 2024-06-18T02:49:15.642 INFO:teuthology.orchestra.run.smithi123.stdout:Upgrade 2 Packages 2024-06-18T02:49:15.643 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:49:15.643 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 58 M 2024-06-18T02:49:15.643 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-06-18T02:49:15.733 INFO:teuthology.orchestra.run.smithi016.stdout:(13/24): ceph-common-17.2.7-1012.g3fe41055.el9. 8.1 MB/s | 22 MB 00:02 2024-06-18T02:49:15.758 INFO:teuthology.orchestra.run.smithi016.stdout:(14/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 55 kB/s | 45 kB 00:00 2024-06-18T02:49:15.783 INFO:teuthology.orchestra.run.smithi016.stdout:(15/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 185 kB/s | 160 kB 00:00 2024-06-18T02:49:16.433 INFO:teuthology.orchestra.run.smithi016.stdout:(16/24): python3-prettytable-0.7.2-27.el9.noarc 65 kB/s | 42 kB 00:00 2024-06-18T02:49:16.509 INFO:teuthology.orchestra.run.smithi016.stdout:(17/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 4.0 MB/s | 309 kB 00:00 2024-06-18T02:49:16.534 INFO:teuthology.orchestra.run.smithi016.stdout:(18/24): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-06-18T02:49:16.559 INFO:teuthology.orchestra.run.smithi016.stdout:(19/24): libunwind-1.6.2-1.el9.x86_64.rpm 2.6 MB/s | 67 kB 00:00 2024-06-18T02:49:16.618 INFO:teuthology.orchestra.run.smithi016.stdout:(20/24): thrift-0.15.0-2.el9.x86_64.rpm 27 MB/s | 1.6 MB 00:00 2024-06-18T02:49:16.643 INFO:teuthology.orchestra.run.smithi016.stdout:(21/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 330 kB/s | 292 kB 00:00 2024-06-18T02:49:16.727 INFO:teuthology.orchestra.run.smithi016.stdout:(22/24): librdkafka-1.6.1-102.el9.x86_64.rpm 667 kB/s | 662 kB 00:00 2024-06-18T02:49:17.446 INFO:teuthology.orchestra.run.smithi123.stdout:(1/24): ceph-base-17.2.7-1012.g3fe41055.el9.x86 3.2 MB/s | 5.0 MB 00:01 2024-06-18T02:49:17.505 INFO:teuthology.orchestra.run.smithi123.stdout:(2/24): ceph-selinux-17.2.7-1012.g3fe41055.el9. 431 kB/s | 25 kB 00:00 2024-06-18T02:49:17.518 INFO:teuthology.orchestra.run.smithi016.stdout:(23/24): librbd1-17.2.7-1012.g3fe41055.el9.x86_ 3.4 MB/s | 2.9 MB 00:00 2024-06-18T02:49:17.680 INFO:teuthology.orchestra.run.smithi123.stdout:(3/24): libcephfs2-17.2.7-1012.g3fe41055.el9.x8 3.6 MB/s | 637 kB 00:00 2024-06-18T02:49:17.768 INFO:teuthology.orchestra.run.smithi016.stdout:(24/24): librados2-17.2.7-1012.g3fe41055.el9.x8 2.7 MB/s | 3.1 MB 00:01 2024-06-18T02:49:17.773 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-06-18T02:49:17.773 INFO:teuthology.orchestra.run.smithi016.stdout:Total 11 MB/s | 58 MB 00:05 2024-06-18T02:49:17.836 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-06-18T02:49:17.874 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-06-18T02:49:17.874 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-06-18T02:49:17.931 INFO:teuthology.orchestra.run.smithi123.stdout:(4/24): ceph-common-17.2.7-1012.g3fe41055.el9.x 11 MB/s | 22 MB 00:02 2024-06-18T02:49:17.956 INFO:teuthology.orchestra.run.smithi123.stdout:(5/24): libradosstriper1-17.2.7-1012.g3fe41055. 1.6 MB/s | 458 kB 00:00 2024-06-18T02:49:18.056 INFO:teuthology.orchestra.run.smithi123.stdout:(6/24): python3-ceph-argparse-17.2.7-1012.g3fe4 450 kB/s | 45 kB 00:00 2024-06-18T02:49:18.148 INFO:teuthology.orchestra.run.smithi123.stdout:(7/24): python3-ceph-common-17.2.7-1012.g3fe410 1.3 MB/s | 118 kB 00:00 2024-06-18T02:49:18.274 INFO:teuthology.orchestra.run.smithi123.stdout:(8/24): librgw2-17.2.7-1012.g3fe41055.el9.x86_6 14 MB/s | 4.9 MB 00:00 2024-06-18T02:49:18.307 INFO:teuthology.orchestra.run.smithi123.stdout:(9/24): python3-cephfs-17.2.7-1012.g3fe41055.el 984 kB/s | 156 kB 00:00 2024-06-18T02:49:18.341 INFO:teuthology.orchestra.run.smithi123.stdout:(10/24): python3-rados-17.2.7-1012.g3fe41055.el 4.7 MB/s | 320 kB 00:00 2024-06-18T02:49:18.383 INFO:teuthology.orchestra.run.smithi123.stdout:(11/24): python3-rgw-17.2.7-1012.g3fe41055.el9. 2.3 MB/s | 99 kB 00:00 2024-06-18T02:49:18.408 INFO:teuthology.orchestra.run.smithi123.stdout:(12/24): python3-rbd-17.2.7-1012.g3fe41055.el9. 2.8 MB/s | 293 kB 00:00 2024-06-18T02:49:18.430 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-06-18T02:49:18.430 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-06-18T02:49:18.550 INFO:teuthology.orchestra.run.smithi123.stdout:(13/24): ceph-radosgw-17.2.7-1012.g3fe41055.el9 5.4 MB/s | 14 MB 00:02 2024-06-18T02:49:18.958 INFO:teuthology.orchestra.run.smithi123.stdout:(14/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 82 kB/s | 45 kB 00:00 2024-06-18T02:49:19.042 INFO:teuthology.orchestra.run.smithi123.stdout:(15/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 243 kB/s | 160 kB 00:00 2024-06-18T02:49:19.176 INFO:teuthology.orchestra.run.smithi123.stdout:(16/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.3 MB/s | 292 kB 00:00 2024-06-18T02:49:19.201 INFO:teuthology.orchestra.run.smithi123.stdout:(17/24): python3-prettytable-0.7.2-27.el9.noarc 266 kB/s | 42 kB 00:00 2024-06-18T02:49:19.285 INFO:teuthology.orchestra.run.smithi123.stdout:(18/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 2.8 MB/s | 309 kB 00:00 2024-06-18T02:49:19.310 INFO:teuthology.orchestra.run.smithi123.stdout:(19/24): libunwind-1.6.2-1.el9.x86_64.rpm 2.9 MB/s | 67 kB 00:00 2024-06-18T02:49:19.336 INFO:teuthology.orchestra.run.smithi123.stdout:(20/24): liboath-2.6.7-2.el9.x86_64.rpm 362 kB/s | 49 kB 00:00 2024-06-18T02:49:19.360 INFO:teuthology.orchestra.run.smithi123.stdout:(21/24): librdkafka-1.6.1-102.el9.x86_64.rpm 870 kB/s | 662 kB 00:00 2024-06-18T02:49:19.411 INFO:teuthology.orchestra.run.smithi123.stdout:(22/24): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-06-18T02:49:19.619 INFO:teuthology.orchestra.run.smithi123.stdout:(23/24): librbd1-17.2.7-1012.g3fe41055.el9.x86_ 11 MB/s | 2.9 MB 00:00 2024-06-18T02:49:19.660 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-06-18T02:49:19.844 INFO:teuthology.orchestra.run.smithi123.stdout:(24/24): librados2-17.2.7-1012.g3fe41055.el9.x8 6.0 MB/s | 3.1 MB 00:00 2024-06-18T02:49:19.851 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-06-18T02:49:19.851 INFO:teuthology.orchestra.run.smithi123.stdout:Total 14 MB/s | 58 MB 00:04 2024-06-18T02:49:19.908 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-06-18T02:49:19.940 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-06-18T02:49:19.940 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-06-18T02:49:20.057 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-06-18T02:49:20.129 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : librados2-2:17.2.7-1012.g3fe41055.el9.x86_64 2/26 2024-06-18T02:49:20.222 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librados2-2:17.2.7-1012.g3fe41055.el9.x86_64 2/26 2024-06-18T02:49:20.303 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rados-2:17.2.7-1012.g3fe41055.el9.x86_64 3/26 2024-06-18T02:49:20.331 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libcephfs2-2:17.2.7-1012.g3fe41055.el9.x86_64 4/26 2024-06-18T02:49:20.426 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libcephfs2-2:17.2.7-1012.g3fe41055.el9.x86_64 4/26 2024-06-18T02:49:20.474 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-06-18T02:49:20.475 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-06-18T02:49:20.476 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-06-18T02:49:20.511 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-06-18T02:49:20.968 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-06-18T02:49:20.992 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librgw2-2:17.2.7-1012.g3fe41055.el9.x86_64 8/26 2024-06-18T02:49:21.038 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librgw2-2:17.2.7-1012.g3fe41055.el9.x86_64 8/26 2024-06-18T02:49:21.067 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-06-18T02:49:21.111 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-ceph-argparse-2:17.2.7-1012.g3fe41055.el9. 10/26 2024-06-18T02:49:21.147 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cephfs-2:17.2.7-1012.g3fe41055.el9.x86_64 11/26 2024-06-18T02:49:21.219 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rgw-2:17.2.7-1012.g3fe41055.el9.x86_64 12/26 2024-06-18T02:49:21.245 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libradosstriper1-2:17.2.7-1012.g3fe41055.el9.x86_6 13/26 2024-06-18T02:49:21.311 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1012.g3fe41055.el9.x86_6 13/26 2024-06-18T02:49:21.351 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-06-18T02:49:21.396 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-06-18T02:49:21.428 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-06-18T02:49:21.646 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-06-18T02:49:21.737 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-06-18T02:49:21.782 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : librbd1-2:17.2.7-1012.g3fe41055.el9.x86_64 18/26 2024-06-18T02:49:21.873 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librbd1-2:17.2.7-1012.g3fe41055.el9.x86_64 18/26 2024-06-18T02:49:21.948 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rbd-2:17.2.7-1012.g3fe41055.el9.x86_64 19/26 2024-06-18T02:49:22.005 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-06-18T02:49:22.008 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-ceph-common-2:17.2.7-1012.g3fe41055.el9.x8 20/26 2024-06-18T02:49:22.064 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : librados2-2:17.2.7-1012.g3fe41055.el9.x86_64 2/26 2024-06-18T02:49:22.147 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librados2-2:17.2.7-1012.g3fe41055.el9.x86_64 2/26 2024-06-18T02:49:22.225 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-rados-2:17.2.7-1012.g3fe41055.el9.x86_64 3/26 2024-06-18T02:49:22.256 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libcephfs2-2:17.2.7-1012.g3fe41055.el9.x86_64 4/26 2024-06-18T02:49:22.337 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libcephfs2-2:17.2.7-1012.g3fe41055.el9.x86_64 4/26 2024-06-18T02:49:22.393 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-06-18T02:49:22.428 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-06-18T02:49:22.868 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-06-18T02:49:22.893 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : librgw2-2:17.2.7-1012.g3fe41055.el9.x86_64 8/26 2024-06-18T02:49:22.947 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librgw2-2:17.2.7-1012.g3fe41055.el9.x86_64 8/26 2024-06-18T02:49:22.976 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-06-18T02:49:23.010 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-ceph-argparse-2:17.2.7-1012.g3fe41055.el9. 10/26 2024-06-18T02:49:23.047 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cephfs-2:17.2.7-1012.g3fe41055.el9.x86_64 11/26 2024-06-18T02:49:23.116 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-rgw-2:17.2.7-1012.g3fe41055.el9.x86_64 12/26 2024-06-18T02:49:23.146 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libradosstriper1-2:17.2.7-1012.g3fe41055.el9.x86_6 13/26 2024-06-18T02:49:23.212 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1012.g3fe41055.el9.x86_6 13/26 2024-06-18T02:49:23.252 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-06-18T02:49:23.287 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-06-18T02:49:23.320 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-06-18T02:49:23.614 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-06-18T02:49:23.650 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : librbd1-2:17.2.7-1012.g3fe41055.el9.x86_64 18/26 2024-06-18T02:49:23.723 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librbd1-2:17.2.7-1012.g3fe41055.el9.x86_64 18/26 2024-06-18T02:49:23.797 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-rbd-2:17.2.7-1012.g3fe41055.el9.x86_64 19/26 2024-06-18T02:49:23.828 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-ceph-common-2:17.2.7-1012.g3fe41055.el9.x8 20/26 2024-06-18T02:49:24.497 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-common-2:17.2.7-1012.g3fe41055.el9.x86_64 21/26 2024-06-18T02:49:24.532 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-common-2:17.2.7-1012.g3fe41055.el9.x86_64 21/26 2024-06-18T02:49:25.078 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-common-2:17.2.7-1012.g3fe41055.el9.x86_64 21/26 2024-06-18T02:49:25.111 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-base-2:17.2.7-1012.g3fe41055.el9.x86_64 22/26 2024-06-18T02:49:25.196 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-base-2:17.2.7-1012.g3fe41055.el9.x86_64 22/26 2024-06-18T02:49:25.196 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-06-18T02:49:25.196 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-06-18T02:49:25.196 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:49:25.229 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-selinux-2:17.2.7-1012.g3fe41055.el9.x86_64 23/26 2024-06-18T02:49:26.248 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-common-2:17.2.7-1012.g3fe41055.el9.x86_64 21/26 2024-06-18T02:49:26.284 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-common-2:17.2.7-1012.g3fe41055.el9.x86_64 21/26 2024-06-18T02:49:26.848 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-common-2:17.2.7-1012.g3fe41055.el9.x86_64 21/26 2024-06-18T02:49:26.889 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-base-2:17.2.7-1012.g3fe41055.el9.x86_64 22/26 2024-06-18T02:49:26.994 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-base-2:17.2.7-1012.g3fe41055.el9.x86_64 22/26 2024-06-18T02:49:26.994 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-06-18T02:49:26.994 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-06-18T02:49:26.994 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:49:27.023 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-selinux-2:17.2.7-1012.g3fe41055.el9.x86_64 23/26 2024-06-18T02:49:35.288 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1012.g3fe41055.el9.x86_64 23/26 2024-06-18T02:49:35.288 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /sys 2024-06-18T02:49:35.288 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /proc 2024-06-18T02:49:35.288 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /mnt 2024-06-18T02:49:35.288 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /var/tmp 2024-06-18T02:49:35.288 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /home 2024-06-18T02:49:35.288 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /root 2024-06-18T02:49:35.288 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /tmp 2024-06-18T02:49:35.289 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:49:36.449 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1012.g3fe41055.el9.x86_64 23/26 2024-06-18T02:49:36.449 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /sys 2024-06-18T02:49:36.449 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /proc 2024-06-18T02:49:36.449 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /mnt 2024-06-18T02:49:36.449 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /var/tmp 2024-06-18T02:49:36.449 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /home 2024-06-18T02:49:36.449 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /root 2024-06-18T02:49:36.449 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /tmp 2024-06-18T02:49:36.450 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:49:36.597 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-radosgw-2:17.2.7-1012.g3fe41055.el9.x86_64 24/26 2024-06-18T02:49:36.678 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1012.g3fe41055.el9.x86_64 24/26 2024-06-18T02:49:36.678 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-18T02:49:36.678 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-06-18T02:49:36.678 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-18T02:49:36.679 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-18T02:49:36.679 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:49:36.680 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-06-18T02:49:36.803 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-06-18T02:49:36.803 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-06-18T02:49:37.751 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-radosgw-2:17.2.7-1012.g3fe41055.el9.x86_64 24/26 2024-06-18T02:49:37.812 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1012.g3fe41055.el9.x86_64 24/26 2024-06-18T02:49:37.812 INFO:teuthology.orchestra.run.smithi123.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-18T02:49:37.812 INFO:teuthology.orchestra.run.smithi123.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-06-18T02:49:37.812 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-18T02:49:37.812 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-18T02:49:37.812 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:49:37.814 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-06-18T02:49:37.895 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-06-18T02:49:37.895 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-06-18T02:49:38.086 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-06-18T02:49:38.086 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-base-2:17.2.7-1012.g3fe41055.el9.x86_64 1/26 2024-06-18T02:49:38.086 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-common-2:17.2.7-1012.g3fe41055.el9.x86_64 2/26 2024-06-18T02:49:38.086 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-radosgw-2:17.2.7-1012.g3fe41055.el9.x86_64 3/26 2024-06-18T02:49:38.086 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-selinux-2:17.2.7-1012.g3fe41055.el9.x86_64 4/26 2024-06-18T02:49:38.086 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libcephfs2-2:17.2.7-1012.g3fe41055.el9.x86_64 5/26 2024-06-18T02:49:38.086 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libradosstriper1-2:17.2.7-1012.g3fe41055.el9.x86_6 6/26 2024-06-18T02:49:38.087 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librgw2-2:17.2.7-1012.g3fe41055.el9.x86_64 7/26 2024-06-18T02:49:38.087 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1012.g3fe41055.el9. 8/26 2024-06-18T02:49:38.087 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-ceph-common-2:17.2.7-1012.g3fe41055.el9.x8 9/26 2024-06-18T02:49:38.087 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cephfs-2:17.2.7-1012.g3fe41055.el9.x86_64 10/26 2024-06-18T02:49:38.087 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rados-2:17.2.7-1012.g3fe41055.el9.x86_64 11/26 2024-06-18T02:49:38.087 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rbd-2:17.2.7-1012.g3fe41055.el9.x86_64 12/26 2024-06-18T02:49:38.087 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rgw-2:17.2.7-1012.g3fe41055.el9.x86_64 13/26 2024-06-18T02:49:38.087 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-06-18T02:49:38.087 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-06-18T02:49:38.087 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-06-18T02:49:38.087 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-06-18T02:49:38.087 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-06-18T02:49:38.087 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-06-18T02:49:38.088 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-06-18T02:49:38.088 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-06-18T02:49:38.088 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-06-18T02:49:38.088 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librados2-2:17.2.7-1012.g3fe41055.el9.x86_64 23/26 2024-06-18T02:49:38.088 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-06-18T02:49:38.088 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librbd1-2:17.2.7-1012.g3fe41055.el9.x86_64 25/26 2024-06-18T02:49:38.803 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-06-18T02:49:38.804 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:49:38.804 INFO:teuthology.orchestra.run.smithi016.stdout:Upgraded: 2024-06-18T02:49:38.804 INFO:teuthology.orchestra.run.smithi016.stdout: librados2-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:49:38.804 INFO:teuthology.orchestra.run.smithi016.stdout: librbd1-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:49:38.804 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-06-18T02:49:38.804 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-base-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:49:38.804 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-common-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:49:38.804 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-radosgw-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:49:38.804 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-selinux-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:49:38.804 INFO:teuthology.orchestra.run.smithi016.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-06-18T02:49:38.804 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs2-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:49:38.804 INFO:teuthology.orchestra.run.smithi016.stdout: liboath-2.6.7-2.el9.x86_64 2024-06-18T02:49:38.805 INFO:teuthology.orchestra.run.smithi016.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-06-18T02:49:38.805 INFO:teuthology.orchestra.run.smithi016.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-06-18T02:49:38.805 INFO:teuthology.orchestra.run.smithi016.stdout: libradosstriper1-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:49:38.805 INFO:teuthology.orchestra.run.smithi016.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-06-18T02:49:38.805 INFO:teuthology.orchestra.run.smithi016.stdout: librgw2-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:49:38.805 INFO:teuthology.orchestra.run.smithi016.stdout: libunwind-1.6.2-1.el9.x86_64 2024-06-18T02:49:38.805 INFO:teuthology.orchestra.run.smithi016.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-06-18T02:49:38.805 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-argparse-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:49:38.805 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-common-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:49:38.805 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cephfs-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:49:38.805 INFO:teuthology.orchestra.run.smithi016.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-06-18T02:49:38.805 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rados-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:49:38.805 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rbd-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:49:38.805 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rgw-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:49:38.806 INFO:teuthology.orchestra.run.smithi016.stdout: thrift-0.15.0-2.el9.x86_64 2024-06-18T02:49:38.806 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:49:38.806 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-06-18T02:49:39.115 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-06-18T02:49:39.115 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-base-2:17.2.7-1012.g3fe41055.el9.x86_64 1/26 2024-06-18T02:49:39.115 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-common-2:17.2.7-1012.g3fe41055.el9.x86_64 2/26 2024-06-18T02:49:39.115 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-radosgw-2:17.2.7-1012.g3fe41055.el9.x86_64 3/26 2024-06-18T02:49:39.115 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-selinux-2:17.2.7-1012.g3fe41055.el9.x86_64 4/26 2024-06-18T02:49:39.115 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libcephfs2-2:17.2.7-1012.g3fe41055.el9.x86_64 5/26 2024-06-18T02:49:39.115 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libradosstriper1-2:17.2.7-1012.g3fe41055.el9.x86_6 6/26 2024-06-18T02:49:39.115 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librgw2-2:17.2.7-1012.g3fe41055.el9.x86_64 7/26 2024-06-18T02:49:39.116 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1012.g3fe41055.el9. 8/26 2024-06-18T02:49:39.116 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-ceph-common-2:17.2.7-1012.g3fe41055.el9.x8 9/26 2024-06-18T02:49:39.116 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cephfs-2:17.2.7-1012.g3fe41055.el9.x86_64 10/26 2024-06-18T02:49:39.116 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-rados-2:17.2.7-1012.g3fe41055.el9.x86_64 11/26 2024-06-18T02:49:39.116 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-rbd-2:17.2.7-1012.g3fe41055.el9.x86_64 12/26 2024-06-18T02:49:39.116 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-rgw-2:17.2.7-1012.g3fe41055.el9.x86_64 13/26 2024-06-18T02:49:39.116 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-06-18T02:49:39.116 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-06-18T02:49:39.116 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-06-18T02:49:39.116 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-06-18T02:49:39.116 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-06-18T02:49:39.116 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-06-18T02:49:39.116 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-06-18T02:49:39.116 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-06-18T02:49:39.117 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-06-18T02:49:39.117 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librados2-2:17.2.7-1012.g3fe41055.el9.x86_64 23/26 2024-06-18T02:49:39.117 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-06-18T02:49:39.117 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librbd1-2:17.2.7-1012.g3fe41055.el9.x86_64 25/26 2024-06-18T02:49:39.152 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-test 2024-06-18T02:49:39.743 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:00:29 ago on Tue 18 Jun 2024 02:49:10 AM UTC. 2024-06-18T02:49:39.849 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-06-18T02:49:39.849 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:49:39.849 INFO:teuthology.orchestra.run.smithi123.stdout:Upgraded: 2024-06-18T02:49:39.849 INFO:teuthology.orchestra.run.smithi123.stdout: librados2-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:49:39.849 INFO:teuthology.orchestra.run.smithi123.stdout: librbd1-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:49:39.849 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-06-18T02:49:39.849 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-base-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:49:39.850 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-common-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:49:39.850 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-radosgw-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:49:39.850 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-selinux-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:49:39.850 INFO:teuthology.orchestra.run.smithi123.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-06-18T02:49:39.850 INFO:teuthology.orchestra.run.smithi123.stdout: libcephfs2-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:49:39.850 INFO:teuthology.orchestra.run.smithi123.stdout: liboath-2.6.7-2.el9.x86_64 2024-06-18T02:49:39.850 INFO:teuthology.orchestra.run.smithi123.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-06-18T02:49:39.850 INFO:teuthology.orchestra.run.smithi123.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-06-18T02:49:39.850 INFO:teuthology.orchestra.run.smithi123.stdout: libradosstriper1-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:49:39.850 INFO:teuthology.orchestra.run.smithi123.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-06-18T02:49:39.850 INFO:teuthology.orchestra.run.smithi123.stdout: librgw2-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:49:39.850 INFO:teuthology.orchestra.run.smithi123.stdout: libunwind-1.6.2-1.el9.x86_64 2024-06-18T02:49:39.850 INFO:teuthology.orchestra.run.smithi123.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-06-18T02:49:39.851 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ceph-argparse-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:49:39.851 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ceph-common-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:49:39.851 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cephfs-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:49:39.851 INFO:teuthology.orchestra.run.smithi123.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-06-18T02:49:39.851 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rados-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:49:39.851 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rbd-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:49:39.851 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rgw-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:49:39.851 INFO:teuthology.orchestra.run.smithi123.stdout: thrift-0.15.0-2.el9.x86_64 2024-06-18T02:49:39.851 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:49:39.851 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-06-18T02:49:39.872 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-06-18T02:49:39.873 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-06-18T02:49:39.873 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2024-06-18T02:49:39.873 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-06-18T02:49:39.873 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-06-18T02:49:39.873 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-test x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 48 M 2024-06-18T02:49:39.873 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2024-06-18T02:49:39.873 INFO:teuthology.orchestra.run.smithi016.stdout: libcephsqlite x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 163 k 2024-06-18T02:49:39.873 INFO:teuthology.orchestra.run.smithi016.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-06-18T02:49:39.873 INFO:teuthology.orchestra.run.smithi016.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-06-18T02:49:39.874 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:49:39.874 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-06-18T02:49:39.874 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-06-18T02:49:39.874 INFO:teuthology.orchestra.run.smithi016.stdout:Install 4 Packages 2024-06-18T02:49:39.874 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:49:39.874 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 48 M 2024-06-18T02:49:39.875 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 192 M 2024-06-18T02:49:39.875 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-06-18T02:49:40.174 INFO:teuthology.orchestra.run.smithi016.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.3 MB/s | 304 kB 00:00 2024-06-18T02:49:40.188 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-test 2024-06-18T02:49:40.199 INFO:teuthology.orchestra.run.smithi016.stdout:(2/4): libcephsqlite-17.2.7-1012.g3fe41055.el9. 652 kB/s | 163 kB 00:00 2024-06-18T02:49:40.224 INFO:teuthology.orchestra.run.smithi016.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-06-18T02:49:40.762 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:00:28 ago on Tue 18 Jun 2024 02:49:12 AM UTC. 2024-06-18T02:49:40.886 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-06-18T02:49:40.887 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-06-18T02:49:40.887 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repository Size 2024-06-18T02:49:40.887 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-06-18T02:49:40.887 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-06-18T02:49:40.887 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-test x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 48 M 2024-06-18T02:49:40.887 INFO:teuthology.orchestra.run.smithi123.stdout:Installing dependencies: 2024-06-18T02:49:40.887 INFO:teuthology.orchestra.run.smithi123.stdout: libcephsqlite x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 163 k 2024-06-18T02:49:40.888 INFO:teuthology.orchestra.run.smithi123.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-06-18T02:49:40.888 INFO:teuthology.orchestra.run.smithi123.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-06-18T02:49:40.888 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:49:40.888 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-06-18T02:49:40.888 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-06-18T02:49:40.888 INFO:teuthology.orchestra.run.smithi123.stdout:Install 4 Packages 2024-06-18T02:49:40.888 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:49:40.889 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 48 M 2024-06-18T02:49:40.889 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 192 M 2024-06-18T02:49:40.889 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-06-18T02:49:41.207 INFO:teuthology.orchestra.run.smithi123.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.1 MB/s | 304 kB 00:00 2024-06-18T02:49:41.266 INFO:teuthology.orchestra.run.smithi123.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-06-18T02:49:41.316 INFO:teuthology.orchestra.run.smithi123.stdout:(3/4): libcephsqlite-17.2.7-1012.g3fe41055.el9. 420 kB/s | 163 kB 00:00 2024-06-18T02:49:43.458 INFO:teuthology.orchestra.run.smithi016.stdout:(4/4): ceph-test-17.2.7-1012.g3fe41055.el9.x86_ 14 MB/s | 48 MB 00:03 2024-06-18T02:49:43.460 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-06-18T02:49:43.460 INFO:teuthology.orchestra.run.smithi016.stdout:Total 13 MB/s | 48 MB 00:03 2024-06-18T02:49:43.488 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-06-18T02:49:43.510 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-06-18T02:49:43.510 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-06-18T02:49:43.896 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-06-18T02:49:43.896 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-06-18T02:49:44.424 INFO:teuthology.orchestra.run.smithi123.stdout:(4/4): ceph-test-17.2.7-1012.g3fe41055.el9.x86_ 14 MB/s | 48 MB 00:03 2024-06-18T02:49:44.426 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-06-18T02:49:44.426 INFO:teuthology.orchestra.run.smithi123.stdout:Total 14 MB/s | 48 MB 00:03 2024-06-18T02:49:44.434 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-06-18T02:49:44.454 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-06-18T02:49:44.473 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-06-18T02:49:44.473 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-06-18T02:49:44.520 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-06-18T02:49:44.566 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-06-18T02:49:44.593 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libcephsqlite-2:17.2.7-1012.g3fe41055.el9.x86_64 3/4 2024-06-18T02:49:44.834 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-06-18T02:49:44.835 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-06-18T02:49:45.356 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-06-18T02:49:45.435 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-06-18T02:49:45.482 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-06-18T02:49:45.509 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libcephsqlite-2:17.2.7-1012.g3fe41055.el9.x86_64 3/4 2024-06-18T02:49:49.243 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1012.g3fe41055.el9.x86_64 3/4 2024-06-18T02:49:49.318 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-test-2:17.2.7-1012.g3fe41055.el9.x86_64 4/4 2024-06-18T02:49:50.041 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1012.g3fe41055.el9.x86_64 3/4 2024-06-18T02:49:50.116 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-test-2:17.2.7-1012.g3fe41055.el9.x86_64 4/4 2024-06-18T02:49:51.565 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-test-2:17.2.7-1012.g3fe41055.el9.x86_64 4/4 2024-06-18T02:49:51.565 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-test-2:17.2.7-1012.g3fe41055.el9.x86_64 1/4 2024-06-18T02:49:51.565 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libcephsqlite-2:17.2.7-1012.g3fe41055.el9.x86_64 2/4 2024-06-18T02:49:51.565 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-06-18T02:49:51.838 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-06-18T02:49:51.838 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:49:51.838 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-06-18T02:49:51.838 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-test-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:49:51.838 INFO:teuthology.orchestra.run.smithi016.stdout: libcephsqlite-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:49:51.838 INFO:teuthology.orchestra.run.smithi016.stdout: socat-1.7.4.1-6.el9.x86_64 2024-06-18T02:49:51.838 INFO:teuthology.orchestra.run.smithi016.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-06-18T02:49:51.838 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:49:51.839 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-06-18T02:49:52.099 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph 2024-06-18T02:49:52.295 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-test-2:17.2.7-1012.g3fe41055.el9.x86_64 4/4 2024-06-18T02:49:52.295 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-test-2:17.2.7-1012.g3fe41055.el9.x86_64 1/4 2024-06-18T02:49:52.295 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libcephsqlite-2:17.2.7-1012.g3fe41055.el9.x86_64 2/4 2024-06-18T02:49:52.295 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-06-18T02:49:52.602 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-06-18T02:49:52.602 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:49:52.602 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-06-18T02:49:52.603 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-test-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:49:52.603 INFO:teuthology.orchestra.run.smithi123.stdout: libcephsqlite-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:49:52.603 INFO:teuthology.orchestra.run.smithi123.stdout: socat-1.7.4.1-6.el9.x86_64 2024-06-18T02:49:52.603 INFO:teuthology.orchestra.run.smithi123.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-06-18T02:49:52.603 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:49:52.603 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-06-18T02:49:52.691 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:00:42 ago on Tue 18 Jun 2024 02:49:10 AM UTC. 2024-06-18T02:49:52.826 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-06-18T02:49:52.829 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-06-18T02:49:52.829 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2024-06-18T02:49:52.829 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-06-18T02:49:52.829 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-06-18T02:49:52.829 INFO:teuthology.orchestra.run.smithi016.stdout: ceph x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 6.5 k 2024-06-18T02:49:52.829 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2024-06-18T02:49:52.829 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mds x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 2.1 M 2024-06-18T02:49:52.829 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 1.4 M 2024-06-18T02:49:52.829 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1012.g3fe41055.el9 ceph-noarch 240 k 2024-06-18T02:49:52.829 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mon x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 3.9 M 2024-06-18T02:49:52.830 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-osd x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 15 M 2024-06-18T02:49:52.830 INFO:teuthology.orchestra.run.smithi016.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-06-18T02:49:52.830 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-06-18T02:49:52.830 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-06-18T02:49:52.830 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-06-18T02:49:52.830 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-06-18T02:49:52.830 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-06-18T02:49:52.830 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-06-18T02:49:52.830 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-collections 2024-06-18T02:49:52.830 INFO:teuthology.orchestra.run.smithi016.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-06-18T02:49:52.830 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-06-18T02:49:52.831 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-06-18T02:49:52.831 INFO:teuthology.orchestra.run.smithi016.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-06-18T02:49:52.831 INFO:teuthology.orchestra.run.smithi016.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-06-18T02:49:52.831 INFO:teuthology.orchestra.run.smithi016.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-06-18T02:49:52.831 INFO:teuthology.orchestra.run.smithi016.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-06-18T02:49:52.831 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-06-18T02:49:52.831 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-06-18T02:49:52.831 INFO:teuthology.orchestra.run.smithi016.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-06-18T02:49:52.831 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-06-18T02:49:52.831 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-06-18T02:49:52.831 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-06-18T02:49:52.831 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-06-18T02:49:52.831 INFO:teuthology.orchestra.run.smithi016.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-06-18T02:49:52.831 INFO:teuthology.orchestra.run.smithi016.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-06-18T02:49:52.832 INFO:teuthology.orchestra.run.smithi016.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-06-18T02:49:52.832 INFO:teuthology.orchestra.run.smithi016.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-06-18T02:49:52.832 INFO:teuthology.orchestra.run.smithi016.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-06-18T02:49:52.832 INFO:teuthology.orchestra.run.smithi016.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-06-18T02:49:52.832 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:49:52.832 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-06-18T02:49:52.832 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-06-18T02:49:52.832 INFO:teuthology.orchestra.run.smithi016.stdout:Install 33 Packages 2024-06-18T02:49:52.832 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:49:52.833 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 27 M 2024-06-18T02:49:52.833 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 96 M 2024-06-18T02:49:52.833 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-06-18T02:49:52.899 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph 2024-06-18T02:49:53.468 INFO:teuthology.orchestra.run.smithi016.stdout:(1/33): ceph-17.2.7-1012.g3fe41055.el9.x86_64.r 35 kB/s | 6.5 kB 00:00 2024-06-18T02:49:53.470 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:00:41 ago on Tue 18 Jun 2024 02:49:12 AM UTC. 2024-06-18T02:49:53.599 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-06-18T02:49:53.601 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-06-18T02:49:53.602 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repository Size 2024-06-18T02:49:53.602 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-06-18T02:49:53.602 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-06-18T02:49:53.602 INFO:teuthology.orchestra.run.smithi123.stdout: ceph x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 6.5 k 2024-06-18T02:49:53.602 INFO:teuthology.orchestra.run.smithi123.stdout:Installing dependencies: 2024-06-18T02:49:53.602 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mds x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 2.1 M 2024-06-18T02:49:53.602 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 1.4 M 2024-06-18T02:49:53.602 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1012.g3fe41055.el9 ceph-noarch 240 k 2024-06-18T02:49:53.602 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mon x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 3.9 M 2024-06-18T02:49:53.602 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-osd x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 15 M 2024-06-18T02:49:53.602 INFO:teuthology.orchestra.run.smithi123.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-06-18T02:49:53.602 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-06-18T02:49:53.603 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-06-18T02:49:53.603 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-06-18T02:49:53.603 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-06-18T02:49:53.603 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-06-18T02:49:53.603 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-06-18T02:49:53.603 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-collections 2024-06-18T02:49:53.603 INFO:teuthology.orchestra.run.smithi123.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-06-18T02:49:53.603 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-06-18T02:49:53.603 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-06-18T02:49:53.603 INFO:teuthology.orchestra.run.smithi123.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-06-18T02:49:53.603 INFO:teuthology.orchestra.run.smithi123.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-06-18T02:49:53.603 INFO:teuthology.orchestra.run.smithi123.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-06-18T02:49:53.603 INFO:teuthology.orchestra.run.smithi123.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-06-18T02:49:53.603 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-06-18T02:49:53.603 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-06-18T02:49:53.604 INFO:teuthology.orchestra.run.smithi123.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-06-18T02:49:53.604 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-06-18T02:49:53.604 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-06-18T02:49:53.604 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-06-18T02:49:53.604 INFO:teuthology.orchestra.run.smithi123.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-06-18T02:49:53.604 INFO:teuthology.orchestra.run.smithi123.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-06-18T02:49:53.604 INFO:teuthology.orchestra.run.smithi123.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-06-18T02:49:53.604 INFO:teuthology.orchestra.run.smithi123.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-06-18T02:49:53.604 INFO:teuthology.orchestra.run.smithi123.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-06-18T02:49:53.604 INFO:teuthology.orchestra.run.smithi123.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-06-18T02:49:53.604 INFO:teuthology.orchestra.run.smithi123.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-06-18T02:49:53.604 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:49:53.604 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-06-18T02:49:53.604 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-06-18T02:49:53.605 INFO:teuthology.orchestra.run.smithi123.stdout:Install 33 Packages 2024-06-18T02:49:53.605 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:49:53.605 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 27 M 2024-06-18T02:49:53.605 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 96 M 2024-06-18T02:49:53.605 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-06-18T02:49:53.684 INFO:teuthology.orchestra.run.smithi016.stdout:(2/33): ceph-mds-17.2.7-1012.g3fe41055.el9.x86_ 5.1 MB/s | 2.1 MB 00:00 2024-06-18T02:49:53.718 INFO:teuthology.orchestra.run.smithi016.stdout:(3/33): ceph-mgr-17.2.7-1012.g3fe41055.el9.x86_ 3.2 MB/s | 1.4 MB 00:00 2024-06-18T02:49:53.810 INFO:teuthology.orchestra.run.smithi016.stdout:(4/33): ceph-mgr-modules-core-17.2.7-1012.g3fe4 2.6 MB/s | 240 kB 00:00 2024-06-18T02:49:54.266 INFO:teuthology.orchestra.run.smithi123.stdout:(1/33): ceph-17.2.7-1012.g3fe41055.el9.x86_64.r 21 kB/s | 6.5 kB 00:00 2024-06-18T02:49:54.469 INFO:teuthology.orchestra.run.smithi016.stdout:(5/33): ceph-osd-17.2.7-1012.g3fe41055.el9.x86_ 20 MB/s | 15 MB 00:00 2024-06-18T02:49:54.494 INFO:teuthology.orchestra.run.smithi016.stdout:(6/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 370 kB/s | 253 kB 00:00 2024-06-18T02:49:54.675 INFO:teuthology.orchestra.run.smithi123.stdout:(2/33): ceph-mon-17.2.7-1012.g3fe41055.el9.x86_ 9.6 MB/s | 3.9 MB 00:00 2024-06-18T02:49:54.678 INFO:teuthology.orchestra.run.smithi016.stdout:(7/33): ceph-mon-17.2.7-1012.g3fe41055.el9.x86_ 3.2 MB/s | 3.9 MB 00:01 2024-06-18T02:49:54.750 INFO:teuthology.orchestra.run.smithi123.stdout:(3/33): ceph-mgr-17.2.7-1012.g3fe41055.el9.x86_ 1.8 MB/s | 1.4 MB 00:00 2024-06-18T02:49:54.821 INFO:teuthology.orchestra.run.smithi016.stdout:(8/33): python3-cryptography-36.0.1-4.el9.x86_6 3.5 MB/s | 1.2 MB 00:00 2024-06-18T02:49:54.845 INFO:teuthology.orchestra.run.smithi016.stdout:(9/33): python3-ply-3.11-14.el9.noarch.rpm 304 kB/s | 106 kB 00:00 2024-06-18T02:49:54.850 INFO:teuthology.orchestra.run.smithi123.stdout:(4/33): ceph-mgr-modules-core-17.2.7-1012.g3fe4 2.4 MB/s | 240 kB 00:00 2024-06-18T02:49:54.870 INFO:teuthology.orchestra.run.smithi016.stdout:(10/33): python3-pycparser-2.20-6.el9.noarch.rp 705 kB/s | 135 kB 00:00 2024-06-18T02:49:54.893 INFO:teuthology.orchestra.run.smithi123.stdout:(5/33): ceph-mds-17.2.7-1012.g3fe41055.el9.x86_ 2.2 MB/s | 2.1 MB 00:00 2024-06-18T02:49:54.929 INFO:teuthology.orchestra.run.smithi016.stdout:(11/33): python3-requests-2.25.1-8.el9.noarch.r 1.1 MB/s | 125 kB 00:00 2024-06-18T02:49:54.979 INFO:teuthology.orchestra.run.smithi016.stdout:(12/33): python3-urllib3-1.26.5-5.el9.noarch.rp 1.9 MB/s | 215 kB 00:00 2024-06-18T02:49:55.060 INFO:teuthology.orchestra.run.smithi123.stdout:(6/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.2 MB/s | 253 kB 00:00 2024-06-18T02:49:55.118 INFO:teuthology.orchestra.run.smithi123.stdout:(7/33): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-06-18T02:49:55.177 INFO:teuthology.orchestra.run.smithi123.stdout:(8/33): python3-pycparser-2.20-6.el9.noarch.rpm 2.3 MB/s | 135 kB 00:00 2024-06-18T02:49:55.235 INFO:teuthology.orchestra.run.smithi123.stdout:(9/33): python3-cryptography-36.0.1-4.el9.x86_6 3.6 MB/s | 1.2 MB 00:00 2024-06-18T02:49:55.260 INFO:teuthology.orchestra.run.smithi123.stdout:(10/33): python3-requests-2.25.1-8.el9.noarch.r 1.5 MB/s | 125 kB 00:00 2024-06-18T02:49:55.411 INFO:teuthology.orchestra.run.smithi123.stdout:(11/33): ceph-osd-17.2.7-1012.g3fe41055.el9.x86 21 MB/s | 15 MB 00:00 2024-06-18T02:49:55.436 INFO:teuthology.orchestra.run.smithi123.stdout:(12/33): python3-urllib3-1.26.5-5.el9.noarch.rp 1.0 MB/s | 215 kB 00:00 2024-06-18T02:49:55.563 INFO:teuthology.orchestra.run.smithi016.stdout:(13/33): python3-markupsafe-1.1.1-12.el9.x86_64 55 kB/s | 35 kB 00:00 2024-06-18T02:49:55.663 INFO:teuthology.orchestra.run.smithi016.stdout:(14/33): python3-pytz-2021.1-5.el9.noarch.rpm 75 kB/s | 51 kB 00:00 2024-06-18T02:49:55.688 INFO:teuthology.orchestra.run.smithi016.stdout:(15/33): python3-mako-1.1.4-6.el9.noarch.rpm 210 kB/s | 172 kB 00:00 2024-06-18T02:49:55.703 INFO:teuthology.orchestra.run.smithi123.stdout:(13/33): python3-markupsafe-1.1.1-12.el9.x86_64 119 kB/s | 35 kB 00:00 2024-06-18T02:49:55.778 INFO:teuthology.orchestra.run.smithi123.stdout:(14/33): python3-pytz-2021.1-5.el9.noarch.rpm 149 kB/s | 51 kB 00:00 2024-06-18T02:49:55.780 INFO:teuthology.orchestra.run.smithi016.stdout:(16/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.7 MB/s | 172 kB 00:00 2024-06-18T02:49:55.805 INFO:teuthology.orchestra.run.smithi016.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 307 kB/s | 43 kB 00:00 2024-06-18T02:49:55.831 INFO:teuthology.orchestra.run.smithi016.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 430 kB/s | 11 kB 00:00 2024-06-18T02:49:55.837 INFO:teuthology.orchestra.run.smithi123.stdout:(15/33): python3-mako-1.1.4-6.el9.noarch.rpm 299 kB/s | 172 kB 00:00 2024-06-18T02:49:55.887 INFO:teuthology.orchestra.run.smithi123.stdout:(16/33): python3-toml-0.10.2-6.el9.noarch.rpm 226 kB/s | 42 kB 00:00 2024-06-18T02:49:55.889 INFO:teuthology.orchestra.run.smithi016.stdout:(19/33): python3-cherrypy-18.6.1-2.el9.noarch.r 3.2 MB/s | 358 kB 00:00 2024-06-18T02:49:55.912 INFO:teuthology.orchestra.run.smithi123.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 325 kB/s | 43 kB 00:00 2024-06-18T02:49:55.914 INFO:teuthology.orchestra.run.smithi016.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 312 kB/s | 18 kB 00:00 2024-06-18T02:49:55.940 INFO:teuthology.orchestra.run.smithi016.stdout:(21/33): python3-jaraco-functools-3.5.0-2.el9.n 782 kB/s | 19 kB 00:00 2024-06-18T02:49:55.965 INFO:teuthology.orchestra.run.smithi016.stdout:(22/33): python3-jaraco-collections-3.0.0-8.el9 461 kB/s | 23 kB 00:00 2024-06-18T02:49:55.971 INFO:teuthology.orchestra.run.smithi123.stdout:(18/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-06-18T02:49:55.990 INFO:teuthology.orchestra.run.smithi016.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 399 kB/s | 20 kB 00:00 2024-06-18T02:49:55.996 INFO:teuthology.orchestra.run.smithi123.stdout:(19/33): python3-cherrypy-18.6.1-2.el9.noarch.r 4.3 MB/s | 358 kB 00:00 2024-06-18T02:49:56.015 INFO:teuthology.orchestra.run.smithi016.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 920 kB/s | 46 kB 00:00 2024-06-18T02:49:56.021 INFO:teuthology.orchestra.run.smithi123.stdout:(20/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 100 kB/s | 11 kB 00:00 2024-06-18T02:49:56.041 INFO:teuthology.orchestra.run.smithi016.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-06-18T02:49:56.046 INFO:teuthology.orchestra.run.smithi123.stdout:(21/33): python3-jaraco-classes-3.2.1-5.el9.noa 241 kB/s | 18 kB 00:00 2024-06-18T02:49:56.066 INFO:teuthology.orchestra.run.smithi016.stdout:(26/33): python3-portend-3.1.0-2.el9.noarch.rpm 662 kB/s | 16 kB 00:00 2024-06-18T02:49:56.072 INFO:teuthology.orchestra.run.smithi123.stdout:(22/33): python3-jaraco-collections-3.0.0-8.el9 307 kB/s | 23 kB 00:00 2024-06-18T02:49:56.091 INFO:teuthology.orchestra.run.smithi016.stdout:(27/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 3.5 MB/s | 90 kB 00:00 2024-06-18T02:49:56.097 INFO:teuthology.orchestra.run.smithi123.stdout:(23/33): python3-jaraco-functools-3.5.0-2.el9.n 258 kB/s | 19 kB 00:00 2024-06-18T02:49:56.116 INFO:teuthology.orchestra.run.smithi016.stdout:(28/33): python3-pecan-1.4.2-3.el9.noarch.rpm 2.6 MB/s | 272 kB 00:00 2024-06-18T02:49:56.122 INFO:teuthology.orchestra.run.smithi123.stdout:(24/33): python3-jaraco-text-3.2.0-6.el9.noarch 261 kB/s | 20 kB 00:00 2024-06-18T02:49:56.142 INFO:teuthology.orchestra.run.smithi016.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 712 kB/s | 36 kB 00:00 2024-06-18T02:49:56.147 INFO:teuthology.orchestra.run.smithi123.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 924 kB/s | 46 kB 00:00 2024-06-18T02:49:56.167 INFO:teuthology.orchestra.run.smithi016.stdout:(30/33): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-06-18T02:49:56.173 INFO:teuthology.orchestra.run.smithi123.stdout:(26/33): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-06-18T02:49:56.192 INFO:teuthology.orchestra.run.smithi016.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 806 kB/s | 20 kB 00:00 2024-06-18T02:49:56.198 INFO:teuthology.orchestra.run.smithi123.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 326 kB/s | 16 kB 00:00 2024-06-18T02:49:56.223 INFO:teuthology.orchestra.run.smithi123.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-06-18T02:49:56.226 INFO:teuthology.orchestra.run.smithi016.stdout:(32/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.0 MB/s | 427 kB 00:00 2024-06-18T02:49:56.249 INFO:teuthology.orchestra.run.smithi123.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 715 kB/s | 36 kB 00:00 2024-06-18T02:49:56.268 INFO:teuthology.orchestra.run.smithi016.stdout:(33/33): python3-toml-0.10.2-6.el9.noarch.rpm 59 kB/s | 42 kB 00:00 2024-06-18T02:49:56.274 INFO:teuthology.orchestra.run.smithi123.stdout:(30/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-06-18T02:49:56.274 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-06-18T02:49:56.275 INFO:teuthology.orchestra.run.smithi016.stdout:Total 7.9 MB/s | 27 MB 00:03 2024-06-18T02:49:56.299 INFO:teuthology.orchestra.run.smithi123.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 802 kB/s | 20 kB 00:00 2024-06-18T02:49:56.324 INFO:teuthology.orchestra.run.smithi123.stdout:(32/33): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-06-18T02:49:56.350 INFO:teuthology.orchestra.run.smithi123.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.1 MB/s | 427 kB 00:00 2024-06-18T02:49:56.354 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-06-18T02:49:56.354 INFO:teuthology.orchestra.run.smithi123.stdout:Total 9.9 MB/s | 27 MB 00:02 2024-06-18T02:49:56.400 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-06-18T02:49:56.441 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-06-18T02:49:56.441 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-06-18T02:49:56.472 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-06-18T02:49:56.508 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-06-18T02:49:56.508 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-06-18T02:49:56.813 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-06-18T02:49:56.814 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-06-18T02:49:56.888 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-06-18T02:49:56.888 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-06-18T02:49:57.992 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-06-18T02:49:58.034 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-06-18T02:49:58.054 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-06-18T02:49:58.087 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-06-18T02:49:58.094 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-06-18T02:49:58.120 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-06-18T02:49:58.126 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-06-18T02:49:58.159 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-06-18T02:49:58.216 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-06-18T02:49:58.251 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-06-18T02:49:58.310 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-06-18T02:49:58.329 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-06-18T02:49:58.356 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-06-18T02:49:58.369 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-06-18T02:49:58.400 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-06-18T02:49:58.405 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-06-18T02:49:58.429 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-06-18T02:49:58.435 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-06-18T02:49:58.463 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-06-18T02:49:58.478 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-06-18T02:49:58.497 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-06-18T02:49:58.511 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-06-18T02:49:58.538 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-06-18T02:49:58.562 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-06-18T02:49:58.581 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-06-18T02:49:58.596 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-06-18T02:49:58.615 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-06-18T02:49:58.631 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-06-18T02:49:58.654 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-06-18T02:49:58.669 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-06-18T02:49:58.739 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-06-18T02:49:58.742 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-06-18T02:49:58.889 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-06-18T02:49:58.893 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-06-18T02:49:58.969 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-06-18T02:49:58.982 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-06-18T02:49:59.022 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 18/33 2024-06-18T02:49:59.050 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 18/33 2024-06-18T02:49:59.060 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-06-18T02:49:59.087 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-06-18T02:49:59.109 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-06-18T02:49:59.146 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-06-18T02:49:59.172 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-06-18T02:49:59.216 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-06-18T02:49:59.368 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-06-18T02:49:59.414 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-06-18T02:49:59.416 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-06-18T02:49:59.468 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-06-18T02:49:59.496 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-06-18T02:49:59.550 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-06-18T02:49:59.580 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-06-18T02:49:59.613 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-06-18T02:49:59.647 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-06-18T02:49:59.684 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-06-18T02:49:59.739 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-06-18T02:49:59.810 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-06-18T02:49:59.884 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1012.g3fe41055.el9. 28/33 2024-06-18T02:49:59.924 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-2:17.2.7-1012.g3fe41055.el9.x86_64 29/33 2024-06-18T02:49:59.948 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1012.g3fe41055.el9.x86_64 29/33 2024-06-18T02:49:59.948 INFO:teuthology.orchestra.run.smithi123.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-18T02:49:59.948 INFO:teuthology.orchestra.run.smithi123.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-06-18T02:49:59.948 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-18T02:49:59.948 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-18T02:49:59.948 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:49:59.970 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1012.g3fe41055.el9. 28/33 2024-06-18T02:49:59.994 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-2:17.2.7-1012.g3fe41055.el9.x86_64 29/33 2024-06-18T02:50:00.033 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1012.g3fe41055.el9.x86_64 29/33 2024-06-18T02:50:00.034 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-18T02:50:00.034 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-06-18T02:50:00.034 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-18T02:50:00.034 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-18T02:50:00.034 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:50:01.276 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-osd-2:17.2.7-1012.g3fe41055.el9.x86_64 30/33 2024-06-18T02:50:01.305 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-osd-2:17.2.7-1012.g3fe41055.el9.x86_64 30/33 2024-06-18T02:50:01.306 INFO:teuthology.orchestra.run.smithi123.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-18T02:50:01.306 INFO:teuthology.orchestra.run.smithi123.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-06-18T02:50:01.306 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-18T02:50:01.306 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-18T02:50:01.306 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:50:01.419 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-osd-2:17.2.7-1012.g3fe41055.el9.x86_64 30/33 2024-06-18T02:50:01.449 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-osd-2:17.2.7-1012.g3fe41055.el9.x86_64 30/33 2024-06-18T02:50:01.449 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-18T02:50:01.449 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-06-18T02:50:01.449 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-18T02:50:01.449 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-18T02:50:01.449 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:50:01.693 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mon-2:17.2.7-1012.g3fe41055.el9.x86_64 31/33 2024-06-18T02:50:01.724 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mon-2:17.2.7-1012.g3fe41055.el9.x86_64 31/33 2024-06-18T02:50:01.724 INFO:teuthology.orchestra.run.smithi123.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-18T02:50:01.724 INFO:teuthology.orchestra.run.smithi123.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-06-18T02:50:01.724 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-18T02:50:01.724 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-18T02:50:01.724 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:50:01.846 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mon-2:17.2.7-1012.g3fe41055.el9.x86_64 31/33 2024-06-18T02:50:01.870 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mon-2:17.2.7-1012.g3fe41055.el9.x86_64 31/33 2024-06-18T02:50:01.870 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-18T02:50:01.871 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-06-18T02:50:01.871 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-18T02:50:01.871 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-18T02:50:01.871 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:50:01.927 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mds-2:17.2.7-1012.g3fe41055.el9.x86_64 32/33 2024-06-18T02:50:01.957 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mds-2:17.2.7-1012.g3fe41055.el9.x86_64 32/33 2024-06-18T02:50:01.957 INFO:teuthology.orchestra.run.smithi123.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-18T02:50:01.957 INFO:teuthology.orchestra.run.smithi123.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-06-18T02:50:01.957 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-18T02:50:01.957 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-18T02:50:01.957 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:50:02.025 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-2:17.2.7-1012.g3fe41055.el9.x86_64 33/33 2024-06-18T02:50:02.079 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mds-2:17.2.7-1012.g3fe41055.el9.x86_64 32/33 2024-06-18T02:50:02.104 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mds-2:17.2.7-1012.g3fe41055.el9.x86_64 32/33 2024-06-18T02:50:02.104 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-18T02:50:02.104 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-06-18T02:50:02.104 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-18T02:50:02.105 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-18T02:50:02.105 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:50:02.171 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-2:17.2.7-1012.g3fe41055.el9.x86_64 33/33 2024-06-18T02:50:03.676 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-2:17.2.7-1012.g3fe41055.el9.x86_64 33/33 2024-06-18T02:50:03.676 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-2:17.2.7-1012.g3fe41055.el9.x86_64 1/33 2024-06-18T02:50:03.676 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mds-2:17.2.7-1012.g3fe41055.el9.x86_64 2/33 2024-06-18T02:50:03.676 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-2:17.2.7-1012.g3fe41055.el9.x86_64 3/33 2024-06-18T02:50:03.676 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mon-2:17.2.7-1012.g3fe41055.el9.x86_64 4/33 2024-06-18T02:50:03.676 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-osd-2:17.2.7-1012.g3fe41055.el9.x86_64 5/33 2024-06-18T02:50:03.677 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1012.g3fe41055.el9. 6/33 2024-06-18T02:50:03.677 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-06-18T02:50:03.677 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-06-18T02:50:03.677 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-06-18T02:50:03.677 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-06-18T02:50:03.677 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-06-18T02:50:03.677 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/33 2024-06-18T02:50:03.677 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-06-18T02:50:03.677 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-06-18T02:50:03.677 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-06-18T02:50:03.677 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-06-18T02:50:03.677 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-06-18T02:50:03.677 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-06-18T02:50:03.677 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-06-18T02:50:03.678 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-06-18T02:50:03.678 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-06-18T02:50:03.678 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-06-18T02:50:03.678 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-06-18T02:50:03.678 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-06-18T02:50:03.678 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-06-18T02:50:03.679 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-06-18T02:50:03.679 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-06-18T02:50:03.679 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-06-18T02:50:03.679 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-06-18T02:50:03.679 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-06-18T02:50:03.679 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-06-18T02:50:03.679 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-06-18T02:50:03.920 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-2:17.2.7-1012.g3fe41055.el9.x86_64 33/33 2024-06-18T02:50:03.920 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-2:17.2.7-1012.g3fe41055.el9.x86_64 1/33 2024-06-18T02:50:03.920 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mds-2:17.2.7-1012.g3fe41055.el9.x86_64 2/33 2024-06-18T02:50:03.920 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-2:17.2.7-1012.g3fe41055.el9.x86_64 3/33 2024-06-18T02:50:03.921 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mon-2:17.2.7-1012.g3fe41055.el9.x86_64 4/33 2024-06-18T02:50:03.921 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-osd-2:17.2.7-1012.g3fe41055.el9.x86_64 5/33 2024-06-18T02:50:03.921 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1012.g3fe41055.el9. 6/33 2024-06-18T02:50:03.921 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-06-18T02:50:03.921 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-06-18T02:50:03.921 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-06-18T02:50:03.921 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-06-18T02:50:03.921 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-06-18T02:50:03.921 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/33 2024-06-18T02:50:03.921 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-06-18T02:50:03.921 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-06-18T02:50:03.921 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-06-18T02:50:03.921 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-06-18T02:50:03.921 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-06-18T02:50:03.922 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-06-18T02:50:03.922 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-06-18T02:50:03.922 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-06-18T02:50:03.922 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-06-18T02:50:03.922 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-06-18T02:50:03.922 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-06-18T02:50:03.922 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-06-18T02:50:03.922 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-06-18T02:50:03.922 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-06-18T02:50:03.922 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-06-18T02:50:03.923 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-06-18T02:50:03.923 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-06-18T02:50:03.923 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-06-18T02:50:03.923 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-06-18T02:50:03.923 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-06-18T02:50:04.419 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-06-18T02:50:04.419 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:50:04.419 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-06-18T02:50:04.419 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:50:04.419 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mds-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:50:04.419 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:50:04.420 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-modules-core-2:17.2.7-1012.g3fe41055.el9.noarch 2024-06-18T02:50:04.420 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mon-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:50:04.420 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-osd-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:50:04.420 INFO:teuthology.orchestra.run.smithi123.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-06-18T02:50:04.420 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-06-18T02:50:04.420 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-06-18T02:50:04.420 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-06-18T02:50:04.420 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-06-18T02:50:04.420 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-06-18T02:50:04.420 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-06-18T02:50:04.420 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-06-18T02:50:04.420 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-06-18T02:50:04.421 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-06-18T02:50:04.421 INFO:teuthology.orchestra.run.smithi123.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-06-18T02:50:04.421 INFO:teuthology.orchestra.run.smithi123.stdout: python3-mako-1.1.4-6.el9.noarch 2024-06-18T02:50:04.421 INFO:teuthology.orchestra.run.smithi123.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-06-18T02:50:04.421 INFO:teuthology.orchestra.run.smithi123.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-06-18T02:50:04.421 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-06-18T02:50:04.421 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ply-3.11-14.el9.noarch 2024-06-18T02:50:04.421 INFO:teuthology.orchestra.run.smithi123.stdout: python3-portend-3.1.0-2.el9.noarch 2024-06-18T02:50:04.421 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-06-18T02:50:04.421 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pycparser-2.20-6.el9.noarch 2024-06-18T02:50:04.421 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pytz-2021.1-5.el9.noarch 2024-06-18T02:50:04.422 INFO:teuthology.orchestra.run.smithi123.stdout: python3-requests-2.25.1-8.el9.noarch 2024-06-18T02:50:04.422 INFO:teuthology.orchestra.run.smithi123.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-06-18T02:50:04.422 INFO:teuthology.orchestra.run.smithi123.stdout: python3-toml-0.10.2-6.el9.noarch 2024-06-18T02:50:04.422 INFO:teuthology.orchestra.run.smithi123.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-06-18T02:50:04.422 INFO:teuthology.orchestra.run.smithi123.stdout: python3-webob-1.8.7-6.el9.noarch 2024-06-18T02:50:04.422 INFO:teuthology.orchestra.run.smithi123.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-06-18T02:50:04.422 INFO:teuthology.orchestra.run.smithi123.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-06-18T02:50:04.422 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:50:04.422 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-06-18T02:50:04.757 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-base 2024-06-18T02:50:04.900 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-06-18T02:50:04.901 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:50:04.901 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-06-18T02:50:04.901 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:50:04.901 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mds-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:50:04.901 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:50:04.902 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-modules-core-2:17.2.7-1012.g3fe41055.el9.noarch 2024-06-18T02:50:04.902 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mon-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:50:04.902 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-osd-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:50:04.902 INFO:teuthology.orchestra.run.smithi016.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-06-18T02:50:04.902 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-06-18T02:50:04.902 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-06-18T02:50:04.902 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-06-18T02:50:04.902 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-06-18T02:50:04.902 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-06-18T02:50:04.902 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-06-18T02:50:04.903 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-06-18T02:50:04.903 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-06-18T02:50:04.903 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-06-18T02:50:04.903 INFO:teuthology.orchestra.run.smithi016.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-06-18T02:50:04.903 INFO:teuthology.orchestra.run.smithi016.stdout: python3-mako-1.1.4-6.el9.noarch 2024-06-18T02:50:04.903 INFO:teuthology.orchestra.run.smithi016.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-06-18T02:50:04.903 INFO:teuthology.orchestra.run.smithi016.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-06-18T02:50:04.903 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-06-18T02:50:04.903 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ply-3.11-14.el9.noarch 2024-06-18T02:50:04.903 INFO:teuthology.orchestra.run.smithi016.stdout: python3-portend-3.1.0-2.el9.noarch 2024-06-18T02:50:04.903 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-06-18T02:50:04.904 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pycparser-2.20-6.el9.noarch 2024-06-18T02:50:04.904 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pytz-2021.1-5.el9.noarch 2024-06-18T02:50:04.904 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests-2.25.1-8.el9.noarch 2024-06-18T02:50:04.904 INFO:teuthology.orchestra.run.smithi016.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-06-18T02:50:04.904 INFO:teuthology.orchestra.run.smithi016.stdout: python3-toml-0.10.2-6.el9.noarch 2024-06-18T02:50:04.904 INFO:teuthology.orchestra.run.smithi016.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-06-18T02:50:04.904 INFO:teuthology.orchestra.run.smithi016.stdout: python3-webob-1.8.7-6.el9.noarch 2024-06-18T02:50:04.904 INFO:teuthology.orchestra.run.smithi016.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-06-18T02:50:04.904 INFO:teuthology.orchestra.run.smithi016.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-06-18T02:50:04.904 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:50:04.904 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-06-18T02:50:05.210 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-base 2024-06-18T02:50:05.341 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:00:53 ago on Tue 18 Jun 2024 02:49:12 AM UTC. 2024-06-18T02:50:05.425 INFO:teuthology.orchestra.run.smithi123.stdout:Package ceph-base-2:17.2.7-1012.g3fe41055.el9.x86_64 is already installed. 2024-06-18T02:50:05.467 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-06-18T02:50:05.468 INFO:teuthology.orchestra.run.smithi123.stdout:Nothing to do. 2024-06-18T02:50:05.468 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-06-18T02:50:05.537 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install cephadm 2024-06-18T02:50:05.833 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:00:55 ago on Tue 18 Jun 2024 02:49:10 AM UTC. 2024-06-18T02:50:05.920 INFO:teuthology.orchestra.run.smithi016.stdout:Package ceph-base-2:17.2.7-1012.g3fe41055.el9.x86_64 is already installed. 2024-06-18T02:50:05.965 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-06-18T02:50:05.966 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2024-06-18T02:50:05.966 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-06-18T02:50:06.032 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install cephadm 2024-06-18T02:50:06.107 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:00:54 ago on Tue 18 Jun 2024 02:49:12 AM UTC. 2024-06-18T02:50:06.235 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-06-18T02:50:06.236 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-06-18T02:50:06.236 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repository Size 2024-06-18T02:50:06.236 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-06-18T02:50:06.236 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-06-18T02:50:06.236 INFO:teuthology.orchestra.run.smithi123.stdout: cephadm noarch 2:17.2.7-1012.g3fe41055.el9 ceph-noarch 82 k 2024-06-18T02:50:06.236 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:50:06.236 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-06-18T02:50:06.236 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-06-18T02:50:06.237 INFO:teuthology.orchestra.run.smithi123.stdout:Install 1 Package 2024-06-18T02:50:06.237 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:50:06.237 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 82 k 2024-06-18T02:50:06.237 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 372 k 2024-06-18T02:50:06.237 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-06-18T02:50:06.459 INFO:teuthology.orchestra.run.smithi123.stdout:cephadm-17.2.7-1012.g3fe41055.el9.noarch.rpm 370 kB/s | 82 kB 00:00 2024-06-18T02:50:06.460 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-06-18T02:50:06.460 INFO:teuthology.orchestra.run.smithi123.stdout:Total 367 kB/s | 82 kB 00:00 2024-06-18T02:50:06.460 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-06-18T02:50:06.465 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-06-18T02:50:06.465 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-06-18T02:50:06.467 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-06-18T02:50:06.467 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-06-18T02:50:06.533 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-06-18T02:50:06.629 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:00:56 ago on Tue 18 Jun 2024 02:49:10 AM UTC. 2024-06-18T02:50:06.761 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-06-18T02:50:06.762 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-06-18T02:50:06.762 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2024-06-18T02:50:06.762 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-06-18T02:50:06.763 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-06-18T02:50:06.763 INFO:teuthology.orchestra.run.smithi016.stdout: cephadm noarch 2:17.2.7-1012.g3fe41055.el9 ceph-noarch 82 k 2024-06-18T02:50:06.763 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:50:06.763 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-06-18T02:50:06.763 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-06-18T02:50:06.763 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2024-06-18T02:50:06.763 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:50:06.763 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 82 k 2024-06-18T02:50:06.763 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 372 k 2024-06-18T02:50:06.763 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-06-18T02:50:07.025 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: cephadm-2:17.2.7-1012.g3fe41055.el9.noarch 1/1 2024-06-18T02:50:07.086 INFO:teuthology.orchestra.run.smithi016.stdout:cephadm-17.2.7-1012.g3fe41055.el9.noarch.rpm 254 kB/s | 82 kB 00:00 2024-06-18T02:50:07.087 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-06-18T02:50:07.087 INFO:teuthology.orchestra.run.smithi016.stdout:Total 253 kB/s | 82 kB 00:00 2024-06-18T02:50:07.087 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-06-18T02:50:07.093 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-06-18T02:50:07.093 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-06-18T02:50:07.095 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-06-18T02:50:07.095 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-06-18T02:50:07.103 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : cephadm-2:17.2.7-1012.g3fe41055.el9.noarch 1/1 2024-06-18T02:50:07.167 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-06-18T02:50:07.457 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: cephadm-2:17.2.7-1012.g3fe41055.el9.noarch 1/1 2024-06-18T02:50:07.631 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: cephadm-2:17.2.7-1012.g3fe41055.el9.noarch 1/1 2024-06-18T02:50:07.689 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : cephadm-2:17.2.7-1012.g3fe41055.el9.noarch 1/1 2024-06-18T02:50:07.689 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:50:07.689 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-06-18T02:50:07.689 INFO:teuthology.orchestra.run.smithi123.stdout: cephadm-2:17.2.7-1012.g3fe41055.el9.noarch 2024-06-18T02:50:07.689 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:50:07.689 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-06-18T02:50:07.728 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : cephadm-2:17.2.7-1012.g3fe41055.el9.noarch 1/1 2024-06-18T02:50:07.914 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-immutable-object-cache 2024-06-18T02:50:08.069 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: cephadm-2:17.2.7-1012.g3fe41055.el9.noarch 1/1 2024-06-18T02:50:08.317 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : cephadm-2:17.2.7-1012.g3fe41055.el9.noarch 1/1 2024-06-18T02:50:08.317 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:50:08.317 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-06-18T02:50:08.317 INFO:teuthology.orchestra.run.smithi016.stdout: cephadm-2:17.2.7-1012.g3fe41055.el9.noarch 2024-06-18T02:50:08.318 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:50:08.318 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-06-18T02:50:08.495 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:00:56 ago on Tue 18 Jun 2024 02:49:12 AM UTC. 2024-06-18T02:50:08.525 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-immutable-object-cache 2024-06-18T02:50:08.624 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-06-18T02:50:08.625 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-06-18T02:50:08.625 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repo Size 2024-06-18T02:50:08.626 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-06-18T02:50:08.626 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-06-18T02:50:08.626 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 137 k 2024-06-18T02:50:08.626 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:50:08.626 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-06-18T02:50:08.626 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-06-18T02:50:08.626 INFO:teuthology.orchestra.run.smithi123.stdout:Install 1 Package 2024-06-18T02:50:08.626 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:50:08.626 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 137 k 2024-06-18T02:50:08.626 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 379 k 2024-06-18T02:50:08.626 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-06-18T02:50:08.973 INFO:teuthology.orchestra.run.smithi123.stdout:ceph-immutable-object-cache-17.2.7-1012.g3fe410 397 kB/s | 137 kB 00:00 2024-06-18T02:50:08.974 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-06-18T02:50:08.974 INFO:teuthology.orchestra.run.smithi123.stdout:Total 394 kB/s | 137 kB 00:00 2024-06-18T02:50:08.974 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-06-18T02:50:08.982 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-06-18T02:50:08.982 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-06-18T02:50:09.039 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-06-18T02:50:09.039 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-06-18T02:50:09.124 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:00:59 ago on Tue 18 Jun 2024 02:49:10 AM UTC. 2024-06-18T02:50:09.191 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-06-18T02:50:09.255 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-06-18T02:50:09.256 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-06-18T02:50:09.256 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repo Size 2024-06-18T02:50:09.256 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-06-18T02:50:09.257 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-06-18T02:50:09.257 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 137 k 2024-06-18T02:50:09.257 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:50:09.257 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-06-18T02:50:09.257 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-06-18T02:50:09.257 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2024-06-18T02:50:09.257 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:50:09.257 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 137 k 2024-06-18T02:50:09.257 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 379 k 2024-06-18T02:50:09.257 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-06-18T02:50:09.282 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1012.g3fe41055. 1/1 2024-06-18T02:50:09.307 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1012.g3fe41055. 1/1 2024-06-18T02:50:09.307 INFO:teuthology.orchestra.run.smithi123.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-18T02:50:09.307 INFO:teuthology.orchestra.run.smithi123.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-06-18T02:50:09.307 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:50:09.485 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-immutable-object-cache-17.2.7-1012.g3fe410 604 kB/s | 137 kB 00:00 2024-06-18T02:50:09.486 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-06-18T02:50:09.486 INFO:teuthology.orchestra.run.smithi016.stdout:Total 599 kB/s | 137 kB 00:00 2024-06-18T02:50:09.486 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-06-18T02:50:09.494 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-06-18T02:50:09.494 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-06-18T02:50:09.555 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-06-18T02:50:09.555 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-06-18T02:50:09.709 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-06-18T02:50:09.778 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1012.g3fe41055. 1/1 2024-06-18T02:50:09.803 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1012.g3fe41055. 1/1 2024-06-18T02:50:09.804 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-18T02:50:09.804 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-06-18T02:50:09.804 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:50:10.151 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1012.g3fe41055. 1/1 2024-06-18T02:50:10.151 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:50:10.151 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-06-18T02:50:10.152 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-immutable-object-cache-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:50:10.152 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:50:10.152 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-06-18T02:50:10.342 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-mgr 2024-06-18T02:50:10.733 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1012.g3fe41055. 1/1 2024-06-18T02:50:10.733 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:50:10.733 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-06-18T02:50:10.733 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-immutable-object-cache-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:50:10.733 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:50:10.734 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-06-18T02:50:10.919 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:00:58 ago on Tue 18 Jun 2024 02:49:12 AM UTC. 2024-06-18T02:50:10.943 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-mgr 2024-06-18T02:50:11.004 INFO:teuthology.orchestra.run.smithi123.stdout:Package ceph-mgr-2:17.2.7-1012.g3fe41055.el9.x86_64 is already installed. 2024-06-18T02:50:11.049 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-06-18T02:50:11.051 INFO:teuthology.orchestra.run.smithi123.stdout:Nothing to do. 2024-06-18T02:50:11.051 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-06-18T02:50:11.117 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-mgr-dashboard 2024-06-18T02:50:11.544 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:01 ago on Tue 18 Jun 2024 02:49:10 AM UTC. 2024-06-18T02:50:11.633 INFO:teuthology.orchestra.run.smithi016.stdout:Package ceph-mgr-2:17.2.7-1012.g3fe41055.el9.x86_64 is already installed. 2024-06-18T02:50:11.675 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:00:59 ago on Tue 18 Jun 2024 02:49:12 AM UTC. 2024-06-18T02:50:11.677 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-06-18T02:50:11.678 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2024-06-18T02:50:11.678 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-06-18T02:50:11.740 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-mgr-dashboard 2024-06-18T02:50:11.805 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-06-18T02:50:11.805 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-06-18T02:50:11.806 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repository Size 2024-06-18T02:50:11.806 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-06-18T02:50:11.806 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-06-18T02:50:11.806 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1012.g3fe41055.el9 ceph-noarch 3.4 M 2024-06-18T02:50:11.806 INFO:teuthology.orchestra.run.smithi123.stdout:Installing dependencies: 2024-06-18T02:50:11.806 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1012.g3fe41055.el9 ceph-noarch 24 k 2024-06-18T02:50:11.806 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1012.g3fe41055.el9 ceph-noarch 15 k 2024-06-18T02:50:11.806 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-06-18T02:50:11.806 INFO:teuthology.orchestra.run.smithi123.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-06-18T02:50:11.806 INFO:teuthology.orchestra.run.smithi123.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-06-18T02:50:11.807 INFO:teuthology.orchestra.run.smithi123.stdout:Installing weak dependencies: 2024-06-18T02:50:11.807 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-06-18T02:50:11.807 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:50:11.807 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-06-18T02:50:11.807 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-06-18T02:50:11.807 INFO:teuthology.orchestra.run.smithi123.stdout:Install 7 Packages 2024-06-18T02:50:11.807 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:50:11.808 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 3.7 M 2024-06-18T02:50:11.808 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 80 M 2024-06-18T02:50:11.808 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-06-18T02:50:12.144 INFO:teuthology.orchestra.run.smithi123.stdout:(1/7): ceph-grafana-dashboards-17.2.7-1012.g3fe 130 kB/s | 24 kB 00:00 2024-06-18T02:50:12.169 INFO:teuthology.orchestra.run.smithi123.stdout:(2/7): ceph-prometheus-alerts-17.2.7-1012.g3fe4 68 kB/s | 15 kB 00:00 2024-06-18T02:50:12.244 INFO:teuthology.orchestra.run.smithi123.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 90 kB/s | 9.0 kB 00:00 2024-06-18T02:50:12.270 INFO:teuthology.orchestra.run.smithi123.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 1.3 MB/s | 31 kB 00:00 2024-06-18T02:50:12.303 INFO:teuthology.orchestra.run.smithi123.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 305 kB/s | 41 kB 00:00 2024-06-18T02:50:12.320 INFO:teuthology.orchestra.run.smithi123.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-06-18T02:50:12.328 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:02 ago on Tue 18 Jun 2024 02:49:10 AM UTC. 2024-06-18T02:50:12.412 INFO:teuthology.orchestra.run.smithi123.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1012.g3fe41055 7.3 MB/s | 3.4 MB 00:00 2024-06-18T02:50:12.416 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-06-18T02:50:12.416 INFO:teuthology.orchestra.run.smithi123.stdout:Total 6.0 MB/s | 3.7 MB 00:00 2024-06-18T02:50:12.417 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-06-18T02:50:12.424 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-06-18T02:50:12.424 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-06-18T02:50:12.460 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-06-18T02:50:12.461 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-06-18T02:50:12.461 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2024-06-18T02:50:12.461 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-06-18T02:50:12.461 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-06-18T02:50:12.461 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1012.g3fe41055.el9 ceph-noarch 3.4 M 2024-06-18T02:50:12.462 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2024-06-18T02:50:12.462 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1012.g3fe41055.el9 ceph-noarch 24 k 2024-06-18T02:50:12.462 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1012.g3fe41055.el9 ceph-noarch 15 k 2024-06-18T02:50:12.462 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-06-18T02:50:12.462 INFO:teuthology.orchestra.run.smithi016.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-06-18T02:50:12.462 INFO:teuthology.orchestra.run.smithi016.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-06-18T02:50:12.462 INFO:teuthology.orchestra.run.smithi016.stdout:Installing weak dependencies: 2024-06-18T02:50:12.462 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-06-18T02:50:12.462 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:50:12.462 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-06-18T02:50:12.462 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-06-18T02:50:12.462 INFO:teuthology.orchestra.run.smithi016.stdout:Install 7 Packages 2024-06-18T02:50:12.462 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:50:12.463 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 3.7 M 2024-06-18T02:50:12.463 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 80 M 2024-06-18T02:50:12.463 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-06-18T02:50:12.485 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-06-18T02:50:12.485 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-06-18T02:50:12.782 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-06-18T02:50:12.830 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-06-18T02:50:12.863 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-06-18T02:50:12.889 INFO:teuthology.orchestra.run.smithi016.stdout:(1/7): ceph-grafana-dashboards-17.2.7-1012.g3fe 128 kB/s | 24 kB 00:00 2024-06-18T02:50:12.910 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-06-18T02:50:12.914 INFO:teuthology.orchestra.run.smithi016.stdout:(2/7): ceph-prometheus-alerts-17.2.7-1012.g3fe4 68 kB/s | 15 kB 00:00 2024-06-18T02:50:12.942 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-06-18T02:50:12.971 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1012.g3fe41055.el9.n 5/7 2024-06-18T02:50:12.998 INFO:teuthology.orchestra.run.smithi016.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 83 kB/s | 9.0 kB 00:00 2024-06-18T02:50:13.023 INFO:teuthology.orchestra.run.smithi016.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 1.3 MB/s | 31 kB 00:00 2024-06-18T02:50:13.048 INFO:teuthology.orchestra.run.smithi016.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 306 kB/s | 41 kB 00:00 2024-06-18T02:50:13.073 INFO:teuthology.orchestra.run.smithi016.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-06-18T02:50:13.199 INFO:teuthology.orchestra.run.smithi016.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1012.g3fe41055 6.7 MB/s | 3.4 MB 00:00 2024-06-18T02:50:13.202 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-06-18T02:50:13.202 INFO:teuthology.orchestra.run.smithi016.stdout:Total 4.9 MB/s | 3.7 MB 00:00 2024-06-18T02:50:13.203 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-06-18T02:50:13.211 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-06-18T02:50:13.212 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-06-18T02:50:13.278 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-06-18T02:50:13.278 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-06-18T02:50:13.623 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-06-18T02:50:13.675 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-06-18T02:50:13.718 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-06-18T02:50:13.766 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-06-18T02:50:13.796 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-06-18T02:50:13.825 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1012.g3fe41055.el9.n 5/7 2024-06-18T02:50:13.958 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1012.g3fe41055.el9. 6/7 2024-06-18T02:50:14.037 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1012.g3fe41055.el9.noarc 7/7 2024-06-18T02:50:14.881 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1012.g3fe41055.el9. 6/7 2024-06-18T02:50:14.920 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1012.g3fe41055.el9.noarc 7/7 2024-06-18T02:50:15.039 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1012.g3fe41055.el9.noarc 7/7 2024-06-18T02:50:15.039 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1012.g3fe41055.el9. 1/7 2024-06-18T02:50:15.039 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1012.g3fe41055.el9.noarc 2/7 2024-06-18T02:50:15.039 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1012.g3fe41055.el9.n 3/7 2024-06-18T02:50:15.040 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-06-18T02:50:15.040 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-06-18T02:50:15.040 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-06-18T02:50:15.349 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-06-18T02:50:15.349 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:50:15.349 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-06-18T02:50:15.349 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-grafana-dashboards-2:17.2.7-1012.g3fe41055.el9.noarch 2024-06-18T02:50:15.349 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-dashboard-2:17.2.7-1012.g3fe41055.el9.noarch 2024-06-18T02:50:15.349 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-prometheus-alerts-2:17.2.7-1012.g3fe41055.el9.noarch 2024-06-18T02:50:15.349 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-06-18T02:50:15.350 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-06-18T02:50:15.350 INFO:teuthology.orchestra.run.smithi123.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-06-18T02:50:15.350 INFO:teuthology.orchestra.run.smithi123.stdout: python3-routes-2.5.1-5.el9.noarch 2024-06-18T02:50:15.350 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:50:15.350 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-06-18T02:50:15.573 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-mgr-diskprediction-local 2024-06-18T02:50:15.924 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1012.g3fe41055.el9.noarc 7/7 2024-06-18T02:50:15.924 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1012.g3fe41055.el9. 1/7 2024-06-18T02:50:15.924 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1012.g3fe41055.el9.noarc 2/7 2024-06-18T02:50:15.924 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1012.g3fe41055.el9.n 3/7 2024-06-18T02:50:15.924 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-06-18T02:50:15.924 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-06-18T02:50:15.924 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-06-18T02:50:16.137 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:01:04 ago on Tue 18 Jun 2024 02:49:12 AM UTC. 2024-06-18T02:50:16.239 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-06-18T02:50:16.239 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:50:16.239 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-06-18T02:50:16.240 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-grafana-dashboards-2:17.2.7-1012.g3fe41055.el9.noarch 2024-06-18T02:50:16.240 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-dashboard-2:17.2.7-1012.g3fe41055.el9.noarch 2024-06-18T02:50:16.240 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-prometheus-alerts-2:17.2.7-1012.g3fe41055.el9.noarch 2024-06-18T02:50:16.240 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-06-18T02:50:16.240 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-06-18T02:50:16.240 INFO:teuthology.orchestra.run.smithi016.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-06-18T02:50:16.240 INFO:teuthology.orchestra.run.smithi016.stdout: python3-routes-2.5.1-5.el9.noarch 2024-06-18T02:50:16.240 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:50:16.240 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-06-18T02:50:16.260 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-06-18T02:50:16.262 INFO:teuthology.orchestra.run.smithi123.stdout:======================================================================================= 2024-06-18T02:50:16.262 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repository Size 2024-06-18T02:50:16.262 INFO:teuthology.orchestra.run.smithi123.stdout:======================================================================================= 2024-06-18T02:50:16.262 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-06-18T02:50:16.262 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1012.g3fe41055.el9 ceph-noarch 7.4 M 2024-06-18T02:50:16.262 INFO:teuthology.orchestra.run.smithi123.stdout:Installing dependencies: 2024-06-18T02:50:16.262 INFO:teuthology.orchestra.run.smithi123.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-06-18T02:50:16.262 INFO:teuthology.orchestra.run.smithi123.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-06-18T02:50:16.263 INFO:teuthology.orchestra.run.smithi123.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-06-18T02:50:16.263 INFO:teuthology.orchestra.run.smithi123.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-06-18T02:50:16.263 INFO:teuthology.orchestra.run.smithi123.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-06-18T02:50:16.263 INFO:teuthology.orchestra.run.smithi123.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-06-18T02:50:16.263 INFO:teuthology.orchestra.run.smithi123.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-06-18T02:50:16.263 INFO:teuthology.orchestra.run.smithi123.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-06-18T02:50:16.263 INFO:teuthology.orchestra.run.smithi123.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-06-18T02:50:16.263 INFO:teuthology.orchestra.run.smithi123.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-06-18T02:50:16.263 INFO:teuthology.orchestra.run.smithi123.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-06-18T02:50:16.263 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:50:16.263 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-06-18T02:50:16.264 INFO:teuthology.orchestra.run.smithi123.stdout:======================================================================================= 2024-06-18T02:50:16.264 INFO:teuthology.orchestra.run.smithi123.stdout:Install 12 Packages 2024-06-18T02:50:16.264 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:50:16.264 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 38 M 2024-06-18T02:50:16.264 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 209 M 2024-06-18T02:50:16.264 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-06-18T02:50:16.453 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-mgr-diskprediction-local 2024-06-18T02:50:17.045 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:07 ago on Tue 18 Jun 2024 02:49:10 AM UTC. 2024-06-18T02:50:17.175 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-06-18T02:50:17.176 INFO:teuthology.orchestra.run.smithi016.stdout:======================================================================================= 2024-06-18T02:50:17.176 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2024-06-18T02:50:17.176 INFO:teuthology.orchestra.run.smithi016.stdout:======================================================================================= 2024-06-18T02:50:17.176 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-06-18T02:50:17.176 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1012.g3fe41055.el9 ceph-noarch 7.4 M 2024-06-18T02:50:17.176 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2024-06-18T02:50:17.177 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-06-18T02:50:17.177 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-06-18T02:50:17.177 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-06-18T02:50:17.177 INFO:teuthology.orchestra.run.smithi016.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-06-18T02:50:17.177 INFO:teuthology.orchestra.run.smithi016.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-06-18T02:50:17.177 INFO:teuthology.orchestra.run.smithi016.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-06-18T02:50:17.177 INFO:teuthology.orchestra.run.smithi016.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-06-18T02:50:17.177 INFO:teuthology.orchestra.run.smithi016.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-06-18T02:50:17.177 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-06-18T02:50:17.177 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-06-18T02:50:17.177 INFO:teuthology.orchestra.run.smithi016.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-06-18T02:50:17.177 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:50:17.178 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-06-18T02:50:17.178 INFO:teuthology.orchestra.run.smithi016.stdout:======================================================================================= 2024-06-18T02:50:17.178 INFO:teuthology.orchestra.run.smithi016.stdout:Install 12 Packages 2024-06-18T02:50:17.178 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:50:17.178 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 38 M 2024-06-18T02:50:17.178 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 209 M 2024-06-18T02:50:17.178 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-06-18T02:50:17.220 INFO:teuthology.orchestra.run.smithi123.stdout:(1/12): ceph-mgr-diskprediction-local-17.2.7-10 12 MB/s | 7.4 MB 00:00 2024-06-18T02:50:17.320 INFO:teuthology.orchestra.run.smithi123.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 327 kB/s | 33 kB 00:00 2024-06-18T02:50:17.503 INFO:teuthology.orchestra.run.smithi123.stdout:(3/12): libquadmath-11.4.1-3.el9.x86_64.rpm 217 kB/s | 195 kB 00:00 2024-06-18T02:50:17.554 INFO:teuthology.orchestra.run.smithi123.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 13 MB/s | 3.0 MB 00:00 2024-06-18T02:50:17.590 INFO:teuthology.orchestra.run.smithi016.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 858 kB/s | 195 kB 00:00 2024-06-18T02:50:17.604 INFO:teuthology.orchestra.run.smithi123.stdout:(5/12): libgfortran-11.4.1-3.el9.x86_64.rpm 801 kB/s | 803 kB 00:01 2024-06-18T02:50:17.629 INFO:teuthology.orchestra.run.smithi123.stdout:(6/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 145 kB/s | 18 kB 00:00 2024-06-18T02:50:17.657 INFO:teuthology.orchestra.run.smithi016.stdout:(2/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.7 MB/s | 803 kB 00:00 2024-06-18T02:50:17.671 INFO:teuthology.orchestra.run.smithi123.stdout:(7/12): openblas-0.3.21-2.el9.x86_64.rpm 301 kB/s | 35 kB 00:00 2024-06-18T02:50:17.696 INFO:teuthology.orchestra.run.smithi123.stdout:(8/12): python3-devel-3.9.18-3.el9.x86_64.rpm 3.6 MB/s | 244 kB 00:00 2024-06-18T02:50:17.699 INFO:teuthology.orchestra.run.smithi016.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 302 kB/s | 33 kB 00:00 2024-06-18T02:50:17.747 INFO:teuthology.orchestra.run.smithi123.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 4.9 MB/s | 250 kB 00:00 2024-06-18T02:50:17.749 INFO:teuthology.orchestra.run.smithi016.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 365 kB/s | 18 kB 00:00 2024-06-18T02:50:17.808 INFO:teuthology.orchestra.run.smithi016.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 603 kB/s | 35 kB 00:00 2024-06-18T02:50:18.105 INFO:teuthology.orchestra.run.smithi123.stdout:(10/12): openblas-openmp-0.3.21-2.el9.x86_64.rp 9.5 MB/s | 4.7 MB 00:00 2024-06-18T02:50:18.258 INFO:teuthology.orchestra.run.smithi016.stdout:(6/12): ceph-mgr-diskprediction-local-17.2.7-10 8.3 MB/s | 7.4 MB 00:00 2024-06-18T02:50:18.272 INFO:teuthology.orchestra.run.smithi123.stdout:(11/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 31 MB/s | 16 MB 00:00 2024-06-18T02:50:18.347 INFO:teuthology.orchestra.run.smithi123.stdout:(12/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 7.5 MB/s | 5.1 MB 00:00 2024-06-18T02:50:18.348 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-06-18T02:50:18.349 INFO:teuthology.orchestra.run.smithi123.stdout:Total 18 MB/s | 38 MB 00:02 2024-06-18T02:50:18.433 INFO:teuthology.orchestra.run.smithi016.stdout:(7/12): python3-devel-3.9.18-3.el9.x86_64.rpm 1.4 MB/s | 244 kB 00:00 2024-06-18T02:50:18.525 INFO:teuthology.orchestra.run.smithi016.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 3.4 MB/s | 3.0 MB 00:00 2024-06-18T02:50:18.609 INFO:teuthology.orchestra.run.smithi016.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.9 MB/s | 250 kB 00:00 2024-06-18T02:50:18.696 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-06-18T02:50:18.722 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-06-18T02:50:18.722 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-06-18T02:50:19.134 INFO:teuthology.orchestra.run.smithi016.stdout:(10/12): openblas-openmp-0.3.21-2.el9.x86_64.rp 3.6 MB/s | 4.7 MB 00:01 2024-06-18T02:50:19.232 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-06-18T02:50:19.232 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-06-18T02:50:19.999 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-06-18T02:50:20.026 INFO:teuthology.orchestra.run.smithi016.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 3.2 MB/s | 5.1 MB 00:01 2024-06-18T02:50:20.073 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-06-18T02:50:20.119 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-06-18T02:50:20.212 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-06-18T02:50:20.255 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-06-18T02:50:20.433 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-06-18T02:50:20.487 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-06-18T02:50:20.571 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-06-18T02:50:21.226 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-06-18T02:50:21.284 INFO:teuthology.orchestra.run.smithi016.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 6.1 MB/s | 16 MB 00:02 2024-06-18T02:50:21.286 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-06-18T02:50:21.286 INFO:teuthology.orchestra.run.smithi016.stdout:Total 9.3 MB/s | 38 MB 00:04 2024-06-18T02:50:21.359 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-06-18T02:50:21.654 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-06-18T02:50:21.681 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-06-18T02:50:21.681 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-06-18T02:50:22.203 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-06-18T02:50:22.203 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-06-18T02:50:22.613 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-06-18T02:50:22.986 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-06-18T02:50:23.050 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-06-18T02:50:23.100 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-06-18T02:50:23.203 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-06-18T02:50:23.242 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-06-18T02:50:23.437 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-06-18T02:50:23.466 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-06-18T02:50:23.546 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-06-18T02:50:23.650 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-06-18T02:50:23.678 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1012.g3fe41 12/12 2024-06-18T02:50:24.225 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-06-18T02:50:24.366 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-06-18T02:50:25.688 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-06-18T02:50:26.326 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1012.g3fe41 12/12 2024-06-18T02:50:26.326 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1012.g3fe41 1/12 2024-06-18T02:50:26.326 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-06-18T02:50:26.326 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-06-18T02:50:26.326 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-06-18T02:50:26.326 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-06-18T02:50:26.327 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-06-18T02:50:26.327 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-06-18T02:50:26.327 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-06-18T02:50:26.327 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-06-18T02:50:26.327 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-06-18T02:50:26.327 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-06-18T02:50:26.737 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-06-18T02:50:26.737 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:50:26.737 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-06-18T02:50:26.737 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1012.g3fe41055.el9.noarch 2024-06-18T02:50:26.738 INFO:teuthology.orchestra.run.smithi123.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-06-18T02:50:26.738 INFO:teuthology.orchestra.run.smithi123.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-06-18T02:50:26.738 INFO:teuthology.orchestra.run.smithi123.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-06-18T02:50:26.738 INFO:teuthology.orchestra.run.smithi123.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-06-18T02:50:26.738 INFO:teuthology.orchestra.run.smithi123.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-06-18T02:50:26.738 INFO:teuthology.orchestra.run.smithi123.stdout: openblas-0.3.21-2.el9.x86_64 2024-06-18T02:50:26.738 INFO:teuthology.orchestra.run.smithi123.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-06-18T02:50:26.738 INFO:teuthology.orchestra.run.smithi123.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-06-18T02:50:26.738 INFO:teuthology.orchestra.run.smithi123.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-06-18T02:50:26.738 INFO:teuthology.orchestra.run.smithi123.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-06-18T02:50:26.738 INFO:teuthology.orchestra.run.smithi123.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-06-18T02:50:26.738 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:50:26.738 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-06-18T02:50:26.787 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-06-18T02:50:26.815 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1012.g3fe41 12/12 2024-06-18T02:50:27.123 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-mgr-rook 2024-06-18T02:50:27.701 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:01:15 ago on Tue 18 Jun 2024 02:49:12 AM UTC. 2024-06-18T02:50:27.827 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-06-18T02:50:27.828 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-06-18T02:50:27.829 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repository Size 2024-06-18T02:50:27.829 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-06-18T02:50:27.829 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-06-18T02:50:27.829 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-rook noarch 2:17.2.7-1012.g3fe41055.el9 ceph-noarch 47 k 2024-06-18T02:50:27.829 INFO:teuthology.orchestra.run.smithi123.stdout:Installing dependencies: 2024-06-18T02:50:27.829 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-06-18T02:50:27.829 INFO:teuthology.orchestra.run.smithi123.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-06-18T02:50:27.829 INFO:teuthology.orchestra.run.smithi123.stdout: python3-google-auth noarch 1:2.30.0-1.el9 epel 213 k 2024-06-18T02:50:27.829 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-06-18T02:50:27.829 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-06-18T02:50:27.829 INFO:teuthology.orchestra.run.smithi123.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-06-18T02:50:27.830 INFO:teuthology.orchestra.run.smithi123.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-06-18T02:50:27.830 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-06-18T02:50:27.830 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-06-18T02:50:27.830 INFO:teuthology.orchestra.run.smithi123.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-06-18T02:50:27.830 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-06-18T02:50:27.830 INFO:teuthology.orchestra.run.smithi123.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-06-18T02:50:27.830 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:50:27.830 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-06-18T02:50:27.830 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-06-18T02:50:27.830 INFO:teuthology.orchestra.run.smithi123.stdout:Install 13 Packages 2024-06-18T02:50:27.830 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:50:27.831 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 2.2 M 2024-06-18T02:50:27.831 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 26 M 2024-06-18T02:50:27.831 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-06-18T02:50:28.286 INFO:teuthology.orchestra.run.smithi123.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 101 kB/s | 19 kB 00:00 2024-06-18T02:50:28.303 INFO:teuthology.orchestra.run.smithi123.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 124 kB/s | 26 kB 00:00 2024-06-18T02:50:28.512 INFO:teuthology.orchestra.run.smithi123.stdout:(3/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 761 kB/s | 159 kB 00:00 2024-06-18T02:50:28.545 INFO:teuthology.orchestra.run.smithi123.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 861 kB/s | 222 kB 00:00 2024-06-18T02:50:28.570 INFO:teuthology.orchestra.run.smithi123.stdout:(5/13): ceph-mgr-rook-17.2.7-1012.g3fe41055.el9 100 kB/s | 47 kB 00:00 2024-06-18T02:50:28.621 INFO:teuthology.orchestra.run.smithi123.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 711 kB/s | 54 kB 00:00 2024-06-18T02:50:28.646 INFO:teuthology.orchestra.run.smithi123.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.0 MB/s | 279 kB 00:00 2024-06-18T02:50:28.696 INFO:teuthology.orchestra.run.smithi123.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 256 kB/s | 32 kB 00:00 2024-06-18T02:50:28.746 INFO:teuthology.orchestra.run.smithi123.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 121 kB/s | 15 kB 00:00 2024-06-18T02:50:28.771 INFO:teuthology.orchestra.run.smithi123.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.5 MB/s | 59 kB 00:00 2024-06-18T02:50:28.797 INFO:teuthology.orchestra.run.smithi123.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.5 MB/s | 90 kB 00:00 2024-06-18T02:50:28.822 INFO:teuthology.orchestra.run.smithi123.stdout:(12/13): python3-google-auth-2.30.0-1.el9.noarc 1.4 MB/s | 213 kB 00:00 2024-06-18T02:50:28.864 INFO:teuthology.orchestra.run.smithi123.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.2 MB/s | 1.0 MB 00:00 2024-06-18T02:50:28.866 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-06-18T02:50:28.867 INFO:teuthology.orchestra.run.smithi123.stdout:Total 2.1 MB/s | 2.2 MB 00:01 2024-06-18T02:50:28.929 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-06-18T02:50:28.943 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-06-18T02:50:28.944 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-06-18T02:50:29.027 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-06-18T02:50:29.028 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-06-18T02:50:29.331 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1012.g3fe41 12/12 2024-06-18T02:50:29.331 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1012.g3fe41 1/12 2024-06-18T02:50:29.331 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-06-18T02:50:29.332 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-06-18T02:50:29.332 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-06-18T02:50:29.332 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-06-18T02:50:29.332 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-06-18T02:50:29.332 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-06-18T02:50:29.332 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-06-18T02:50:29.332 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-06-18T02:50:29.332 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-06-18T02:50:29.332 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-06-18T02:50:29.530 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-06-18T02:50:29.691 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-06-18T02:50:29.747 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-06-18T02:50:29.768 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-06-18T02:50:29.768 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:50:29.768 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-06-18T02:50:29.768 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1012.g3fe41055.el9.noarch 2024-06-18T02:50:29.768 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-06-18T02:50:29.768 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-06-18T02:50:29.768 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-06-18T02:50:29.768 INFO:teuthology.orchestra.run.smithi016.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-06-18T02:50:29.768 INFO:teuthology.orchestra.run.smithi016.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-06-18T02:50:29.768 INFO:teuthology.orchestra.run.smithi016.stdout: openblas-0.3.21-2.el9.x86_64 2024-06-18T02:50:29.768 INFO:teuthology.orchestra.run.smithi016.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-06-18T02:50:29.768 INFO:teuthology.orchestra.run.smithi016.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-06-18T02:50:29.768 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-06-18T02:50:29.769 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-06-18T02:50:29.769 INFO:teuthology.orchestra.run.smithi016.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-06-18T02:50:29.769 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:50:29.769 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-06-18T02:50:29.807 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-06-18T02:50:29.845 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-06-18T02:50:29.886 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-06-18T02:50:29.982 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-06-18T02:50:30.089 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-google-auth-1:2.30.0-1.el9.noarch 7/13 2024-06-18T02:50:30.127 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-mgr-rook 2024-06-18T02:50:30.138 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-06-18T02:50:30.729 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:20 ago on Tue 18 Jun 2024 02:49:10 AM UTC. 2024-06-18T02:50:30.847 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-06-18T02:50:30.861 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-06-18T02:50:30.862 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-06-18T02:50:30.862 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2024-06-18T02:50:30.862 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-06-18T02:50:30.862 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-06-18T02:50:30.862 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-rook noarch 2:17.2.7-1012.g3fe41055.el9 ceph-noarch 47 k 2024-06-18T02:50:30.862 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2024-06-18T02:50:30.862 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-06-18T02:50:30.863 INFO:teuthology.orchestra.run.smithi016.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-06-18T02:50:30.863 INFO:teuthology.orchestra.run.smithi016.stdout: python3-google-auth noarch 1:2.30.0-1.el9 epel 213 k 2024-06-18T02:50:30.863 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-06-18T02:50:30.863 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-06-18T02:50:30.863 INFO:teuthology.orchestra.run.smithi016.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-06-18T02:50:30.863 INFO:teuthology.orchestra.run.smithi016.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-06-18T02:50:30.863 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-06-18T02:50:30.863 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-06-18T02:50:30.863 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-06-18T02:50:30.863 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-06-18T02:50:30.863 INFO:teuthology.orchestra.run.smithi016.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-06-18T02:50:30.863 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:50:30.864 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-06-18T02:50:30.864 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-06-18T02:50:30.864 INFO:teuthology.orchestra.run.smithi016.stdout:Install 13 Packages 2024-06-18T02:50:30.864 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:50:30.864 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 2.2 M 2024-06-18T02:50:30.864 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 26 M 2024-06-18T02:50:30.864 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-06-18T02:50:30.893 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-06-18T02:50:30.927 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-06-18T02:50:30.972 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-06-18T02:50:30.996 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-rook-2:17.2.7-1012.g3fe41055.el9.noarch 13/13 2024-06-18T02:50:31.650 INFO:teuthology.orchestra.run.smithi016.stdout:(1/13): ceph-mgr-rook-17.2.7-1012.g3fe41055.el9 128 kB/s | 47 kB 00:00 2024-06-18T02:50:31.721 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1012.g3fe41055.el9.noarch 13/13 2024-06-18T02:50:31.721 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1012.g3fe41055.el9.noarch 1/13 2024-06-18T02:50:31.721 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-06-18T02:50:31.721 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-06-18T02:50:31.721 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-06-18T02:50:31.721 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-06-18T02:50:31.721 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-06-18T02:50:31.722 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-06-18T02:50:31.722 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-06-18T02:50:31.722 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-06-18T02:50:31.722 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-google-auth-1:2.30.0-1.el9.noarch 10/13 2024-06-18T02:50:31.722 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-06-18T02:50:31.722 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-06-18T02:50:31.892 INFO:teuthology.orchestra.run.smithi016.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 31 kB/s | 19 kB 00:00 2024-06-18T02:50:31.992 INFO:teuthology.orchestra.run.smithi016.stdout:(3/13): python3-jsonpatch-1.21-16.el9.noarch.rp 36 kB/s | 26 kB 00:00 2024-06-18T02:50:32.047 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-06-18T02:50:32.048 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:50:32.048 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-06-18T02:50:32.048 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-rook-2:17.2.7-1012.g3fe41055.el9.noarch 2024-06-18T02:50:32.048 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-06-18T02:50:32.048 INFO:teuthology.orchestra.run.smithi123.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-06-18T02:50:32.048 INFO:teuthology.orchestra.run.smithi123.stdout: python3-google-auth-1:2.30.0-1.el9.noarch 2024-06-18T02:50:32.048 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-06-18T02:50:32.048 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-06-18T02:50:32.048 INFO:teuthology.orchestra.run.smithi123.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-06-18T02:50:32.048 INFO:teuthology.orchestra.run.smithi123.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-06-18T02:50:32.048 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-06-18T02:50:32.049 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-06-18T02:50:32.049 INFO:teuthology.orchestra.run.smithi123.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-06-18T02:50:32.049 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rsa-4.9-2.el9.noarch 2024-06-18T02:50:32.049 INFO:teuthology.orchestra.run.smithi123.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-06-18T02:50:32.049 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:50:32.049 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-06-18T02:50:32.291 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-mgr-cephadm 2024-06-18T02:50:32.576 INFO:teuthology.orchestra.run.smithi016.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 240 kB/s | 222 kB 00:00 2024-06-18T02:50:32.760 INFO:teuthology.orchestra.run.smithi016.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 183 kB/s | 159 kB 00:00 2024-06-18T02:50:32.826 INFO:teuthology.orchestra.run.smithi016.stdout:(6/13): python3-cachetools-4.2.4-1.el9.noarch.r 487 kB/s | 32 kB 00:00 2024-06-18T02:50:32.852 INFO:teuthology.orchestra.run.smithi016.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 604 kB/s | 15 kB 00:00 2024-06-18T02:50:32.891 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:01:20 ago on Tue 18 Jun 2024 02:49:12 AM UTC. 2024-06-18T02:50:32.910 INFO:teuthology.orchestra.run.smithi016.stdout:(8/13): python3-pyasn1-modules-0.4.8-6.el9.noar 304 kB/s | 279 kB 00:00 2024-06-18T02:50:32.961 INFO:teuthology.orchestra.run.smithi016.stdout:(9/13): python3-google-auth-2.30.0-1.el9.noarch 2.6 MB/s | 213 kB 00:00 2024-06-18T02:50:32.986 INFO:teuthology.orchestra.run.smithi016.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.6 MB/s | 59 kB 00:00 2024-06-18T02:50:33.012 INFO:teuthology.orchestra.run.smithi016.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.5 MB/s | 90 kB 00:00 2024-06-18T02:50:33.022 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-06-18T02:50:33.023 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-06-18T02:50:33.023 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repository Size 2024-06-18T02:50:33.023 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-06-18T02:50:33.023 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-06-18T02:50:33.024 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1012.g3fe41055.el9 ceph-noarch 123 k 2024-06-18T02:50:33.024 INFO:teuthology.orchestra.run.smithi123.stdout:Installing dependencies: 2024-06-18T02:50:33.024 INFO:teuthology.orchestra.run.smithi123.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-06-18T02:50:33.024 INFO:teuthology.orchestra.run.smithi123.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-06-18T02:50:33.024 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-06-18T02:50:33.024 INFO:teuthology.orchestra.run.smithi123.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-06-18T02:50:33.024 INFO:teuthology.orchestra.run.smithi123.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-06-18T02:50:33.024 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:50:33.024 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-06-18T02:50:33.024 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-06-18T02:50:33.024 INFO:teuthology.orchestra.run.smithi123.stdout:Install 6 Packages 2024-06-18T02:50:33.024 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:50:33.025 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 7.0 M 2024-06-18T02:50:33.025 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 33 M 2024-06-18T02:50:33.025 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-06-18T02:50:33.054 INFO:teuthology.orchestra.run.smithi016.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 7.1 MB/s | 1.0 MB 00:00 2024-06-18T02:50:33.346 INFO:teuthology.orchestra.run.smithi016.stdout:(13/13): python3-requests-oauthlib-1.3.0-12.el9 70 kB/s | 54 kB 00:00 2024-06-18T02:50:33.349 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-06-18T02:50:33.349 INFO:teuthology.orchestra.run.smithi016.stdout:Total 909 kB/s | 2.2 MB 00:02 2024-06-18T02:50:33.422 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-06-18T02:50:33.438 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-06-18T02:50:33.438 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-06-18T02:50:33.534 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-06-18T02:50:33.534 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-06-18T02:50:33.736 INFO:teuthology.orchestra.run.smithi123.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.3 MB/s | 248 kB 00:00 2024-06-18T02:50:33.761 INFO:teuthology.orchestra.run.smithi123.stdout:(2/6): ceph-mgr-cephadm-17.2.7-1012.g3fe41055.e 563 kB/s | 123 kB 00:00 2024-06-18T02:50:33.845 INFO:teuthology.orchestra.run.smithi123.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 4.9 MB/s | 548 kB 00:00 2024-06-18T02:50:33.870 INFO:teuthology.orchestra.run.smithi123.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 531 kB/s | 58 kB 00:00 2024-06-18T02:50:33.895 INFO:teuthology.orchestra.run.smithi123.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 1.0 MB/s | 51 kB 00:00 2024-06-18T02:50:34.060 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-06-18T02:50:34.246 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-06-18T02:50:34.301 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-06-18T02:50:34.358 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-06-18T02:50:34.402 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-06-18T02:50:34.452 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-06-18T02:50:34.561 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-06-18T02:50:34.669 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-google-auth-1:2.30.0-1.el9.noarch 7/13 2024-06-18T02:50:34.721 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-06-18T02:50:35.387 INFO:teuthology.orchestra.run.smithi123.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 3.2 MB/s | 6.0 MB 00:01 2024-06-18T02:50:35.390 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-06-18T02:50:35.390 INFO:teuthology.orchestra.run.smithi123.stdout:Total 2.9 MB/s | 7.0 MB 00:02 2024-06-18T02:50:35.462 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-06-18T02:50:35.474 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-06-18T02:50:35.474 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-06-18T02:50:35.495 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-06-18T02:50:35.542 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-06-18T02:50:35.571 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-06-18T02:50:35.572 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-06-18T02:50:35.576 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-06-18T02:50:35.622 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-06-18T02:50:35.654 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-rook-2:17.2.7-1012.g3fe41055.el9.noarch 13/13 2024-06-18T02:50:35.886 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-06-18T02:50:36.061 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-06-18T02:50:36.114 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-06-18T02:50:36.389 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1012.g3fe41055.el9.noarch 13/13 2024-06-18T02:50:36.389 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1012.g3fe41055.el9.noarch 1/13 2024-06-18T02:50:36.389 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-06-18T02:50:36.389 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-06-18T02:50:36.389 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-06-18T02:50:36.389 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-06-18T02:50:36.390 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-06-18T02:50:36.390 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-06-18T02:50:36.390 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-06-18T02:50:36.390 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-06-18T02:50:36.390 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-google-auth-1:2.30.0-1.el9.noarch 10/13 2024-06-18T02:50:36.390 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-06-18T02:50:36.390 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-06-18T02:50:36.636 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-06-18T02:50:36.724 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-06-18T02:50:36.780 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-06-18T02:50:36.809 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1012.g3fe41055.el9.noarch 6/6 2024-06-18T02:50:36.847 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-06-18T02:50:36.847 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:50:36.847 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-06-18T02:50:36.847 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-rook-2:17.2.7-1012.g3fe41055.el9.noarch 2024-06-18T02:50:36.847 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-06-18T02:50:36.847 INFO:teuthology.orchestra.run.smithi016.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-06-18T02:50:36.847 INFO:teuthology.orchestra.run.smithi016.stdout: python3-google-auth-1:2.30.0-1.el9.noarch 2024-06-18T02:50:36.848 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-06-18T02:50:36.848 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-06-18T02:50:36.848 INFO:teuthology.orchestra.run.smithi016.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-06-18T02:50:36.848 INFO:teuthology.orchestra.run.smithi016.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-06-18T02:50:36.848 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-06-18T02:50:36.848 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-06-18T02:50:36.848 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-06-18T02:50:36.848 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rsa-4.9-2.el9.noarch 2024-06-18T02:50:36.848 INFO:teuthology.orchestra.run.smithi016.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-06-18T02:50:36.848 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:50:36.848 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-06-18T02:50:37.102 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-mgr-cephadm 2024-06-18T02:50:37.530 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1012.g3fe41055.el9.noarch 6/6 2024-06-18T02:50:37.530 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1012.g3fe41055.el9.noarch 1/6 2024-06-18T02:50:37.530 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-06-18T02:50:37.531 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-06-18T02:50:37.531 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-06-18T02:50:37.531 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-06-18T02:50:37.694 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:27 ago on Tue 18 Jun 2024 02:49:10 AM UTC. 2024-06-18T02:50:37.787 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-06-18T02:50:37.788 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:50:37.788 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-06-18T02:50:37.788 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-cephadm-2:17.2.7-1012.g3fe41055.el9.noarch 2024-06-18T02:50:37.788 INFO:teuthology.orchestra.run.smithi123.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-06-18T02:50:37.788 INFO:teuthology.orchestra.run.smithi123.stdout: python3-babel-2.9.1-2.el9.noarch 2024-06-18T02:50:37.788 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-06-18T02:50:37.788 INFO:teuthology.orchestra.run.smithi123.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-06-18T02:50:37.789 INFO:teuthology.orchestra.run.smithi123.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-06-18T02:50:37.789 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:50:37.789 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-06-18T02:50:37.825 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-06-18T02:50:37.826 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-06-18T02:50:37.826 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2024-06-18T02:50:37.827 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-06-18T02:50:37.827 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-06-18T02:50:37.827 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1012.g3fe41055.el9 ceph-noarch 123 k 2024-06-18T02:50:37.827 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2024-06-18T02:50:37.827 INFO:teuthology.orchestra.run.smithi016.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-06-18T02:50:37.827 INFO:teuthology.orchestra.run.smithi016.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-06-18T02:50:37.827 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-06-18T02:50:37.827 INFO:teuthology.orchestra.run.smithi016.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-06-18T02:50:37.827 INFO:teuthology.orchestra.run.smithi016.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-06-18T02:50:37.827 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:50:37.827 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-06-18T02:50:37.828 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-06-18T02:50:37.828 INFO:teuthology.orchestra.run.smithi016.stdout:Install 6 Packages 2024-06-18T02:50:37.828 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:50:37.828 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 7.0 M 2024-06-18T02:50:37.828 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 33 M 2024-06-18T02:50:37.828 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-06-18T02:50:38.017 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-fuse 2024-06-18T02:50:38.586 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:01:26 ago on Tue 18 Jun 2024 02:49:12 AM UTC. 2024-06-18T02:50:38.713 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-06-18T02:50:38.714 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-06-18T02:50:38.714 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repository Size 2024-06-18T02:50:38.714 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-06-18T02:50:38.714 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-06-18T02:50:38.714 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-fuse x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 812 k 2024-06-18T02:50:38.714 INFO:teuthology.orchestra.run.smithi123.stdout:Installing dependencies: 2024-06-18T02:50:38.714 INFO:teuthology.orchestra.run.smithi123.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-06-18T02:50:38.714 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:50:38.714 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-06-18T02:50:38.714 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-06-18T02:50:38.715 INFO:teuthology.orchestra.run.smithi123.stdout:Install 2 Packages 2024-06-18T02:50:38.715 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:50:38.715 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 892 k 2024-06-18T02:50:38.715 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 2.6 M 2024-06-18T02:50:38.715 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-06-18T02:50:38.919 INFO:teuthology.orchestra.run.smithi016.stdout:(1/6): ceph-mgr-cephadm-17.2.7-1012.g3fe41055.e 236 kB/s | 123 kB 00:00 2024-06-18T02:50:38.972 INFO:teuthology.orchestra.run.smithi123.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 682 kB/s | 80 kB 00:00 2024-06-18T02:50:38.985 INFO:teuthology.orchestra.run.smithi016.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 8.1 MB/s | 548 kB 00:00 2024-06-18T02:50:39.011 INFO:teuthology.orchestra.run.smithi016.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.3 MB/s | 58 kB 00:00 2024-06-18T02:50:39.045 INFO:teuthology.orchestra.run.smithi016.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 1.5 MB/s | 51 kB 00:00 2024-06-18T02:50:39.653 INFO:teuthology.orchestra.run.smithi016.stdout:(5/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 197 kB/s | 248 kB 00:01 2024-06-18T02:50:39.672 INFO:teuthology.orchestra.run.smithi123.stdout:(2/2): ceph-fuse-17.2.7-1012.g3fe41055.el9.x86_ 993 kB/s | 812 kB 00:00 2024-06-18T02:50:39.673 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-06-18T02:50:39.673 INFO:teuthology.orchestra.run.smithi123.stdout:Total 931 kB/s | 892 kB 00:00 2024-06-18T02:50:39.685 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-06-18T02:50:39.708 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-06-18T02:50:39.708 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-06-18T02:50:39.785 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-06-18T02:50:39.786 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-06-18T02:50:39.963 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-06-18T02:50:40.092 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-06-18T02:50:40.153 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-fuse-2:17.2.7-1012.g3fe41055.el9.x86_64 2/2 2024-06-18T02:50:40.823 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1012.g3fe41055.el9.x86_64 2/2 2024-06-18T02:50:40.823 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-fuse-2:17.2.7-1012.g3fe41055.el9.x86_64 1/2 2024-06-18T02:50:41.114 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-06-18T02:50:41.114 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:50:41.114 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-06-18T02:50:41.114 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-fuse-2:17.2.7-1012.g3fe41055.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-06-18T02:50:41.114 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:50:41.114 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-06-18T02:50:41.351 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install librados-devel 2024-06-18T02:50:41.940 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:01:29 ago on Tue 18 Jun 2024 02:49:12 AM UTC. 2024-06-18T02:50:42.069 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-06-18T02:50:42.070 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-06-18T02:50:42.070 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repo Size 2024-06-18T02:50:42.070 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-06-18T02:50:42.070 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-06-18T02:50:42.070 INFO:teuthology.orchestra.run.smithi123.stdout: librados-devel x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 126 k 2024-06-18T02:50:42.070 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:50:42.071 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-06-18T02:50:42.071 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-06-18T02:50:42.071 INFO:teuthology.orchestra.run.smithi123.stdout:Install 1 Package 2024-06-18T02:50:42.071 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:50:42.071 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 126 k 2024-06-18T02:50:42.071 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 448 k 2024-06-18T02:50:42.071 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-06-18T02:50:42.485 INFO:teuthology.orchestra.run.smithi123.stdout:librados-devel-17.2.7-1012.g3fe41055.el9.x86_64 305 kB/s | 126 kB 00:00 2024-06-18T02:50:42.485 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-06-18T02:50:42.486 INFO:teuthology.orchestra.run.smithi123.stdout:Total 304 kB/s | 126 kB 00:00 2024-06-18T02:50:42.486 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-06-18T02:50:42.491 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-06-18T02:50:42.491 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-06-18T02:50:42.554 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-06-18T02:50:42.555 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-06-18T02:50:42.704 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-06-18T02:50:42.798 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : librados-devel-2:17.2.7-1012.g3fe41055.el9.x86_64 1/1 2024-06-18T02:50:43.232 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librados-devel-2:17.2.7-1012.g3fe41055.el9.x86_64 1/1 2024-06-18T02:50:43.490 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librados-devel-2:17.2.7-1012.g3fe41055.el9.x86_64 1/1 2024-06-18T02:50:43.490 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:50:43.490 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-06-18T02:50:43.490 INFO:teuthology.orchestra.run.smithi123.stdout: librados-devel-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:50:43.490 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:50:43.490 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-06-18T02:50:43.746 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install libcephfs2 2024-06-18T02:50:44.328 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:01:32 ago on Tue 18 Jun 2024 02:49:12 AM UTC. 2024-06-18T02:50:44.414 INFO:teuthology.orchestra.run.smithi123.stdout:Package libcephfs2-2:17.2.7-1012.g3fe41055.el9.x86_64 is already installed. 2024-06-18T02:50:44.458 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-06-18T02:50:44.459 INFO:teuthology.orchestra.run.smithi123.stdout:Nothing to do. 2024-06-18T02:50:44.459 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-06-18T02:50:44.522 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install libcephfs-devel 2024-06-18T02:50:45.097 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:01:33 ago on Tue 18 Jun 2024 02:49:12 AM UTC. 2024-06-18T02:50:45.222 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-06-18T02:50:45.223 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-06-18T02:50:45.223 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repo Size 2024-06-18T02:50:45.223 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-06-18T02:50:45.223 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-06-18T02:50:45.223 INFO:teuthology.orchestra.run.smithi123.stdout: libcephfs-devel x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 25 k 2024-06-18T02:50:45.223 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:50:45.223 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-06-18T02:50:45.223 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-06-18T02:50:45.224 INFO:teuthology.orchestra.run.smithi123.stdout:Install 1 Package 2024-06-18T02:50:45.224 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:50:45.224 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 25 k 2024-06-18T02:50:45.224 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 110 k 2024-06-18T02:50:45.224 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-06-18T02:50:45.388 INFO:teuthology.orchestra.run.smithi123.stdout:libcephfs-devel-17.2.7-1012.g3fe41055.el9.x86_6 153 kB/s | 25 kB 00:00 2024-06-18T02:50:45.388 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-06-18T02:50:45.388 INFO:teuthology.orchestra.run.smithi123.stdout:Total 151 kB/s | 25 kB 00:00 2024-06-18T02:50:45.389 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-06-18T02:50:45.392 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-06-18T02:50:45.392 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-06-18T02:50:45.393 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-06-18T02:50:45.394 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-06-18T02:50:45.467 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-06-18T02:50:45.576 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libcephfs-devel-2:17.2.7-1012.g3fe41055.el9.x86_64 1/1 2024-06-18T02:50:45.903 INFO:teuthology.orchestra.run.smithi016.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 813 kB/s | 6.0 MB 00:07 2024-06-18T02:50:45.905 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-06-18T02:50:45.905 INFO:teuthology.orchestra.run.smithi016.stdout:Total 883 kB/s | 7.0 MB 00:08 2024-06-18T02:50:45.941 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1012.g3fe41055.el9.x86_64 1/1 2024-06-18T02:50:45.981 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-06-18T02:50:45.994 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-06-18T02:50:45.994 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-06-18T02:50:46.100 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-06-18T02:50:46.100 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-06-18T02:50:46.160 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libcephfs-devel-2:17.2.7-1012.g3fe41055.el9.x86_64 1/1 2024-06-18T02:50:46.160 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:50:46.160 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-06-18T02:50:46.160 INFO:teuthology.orchestra.run.smithi123.stdout: libcephfs-devel-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:50:46.160 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:50:46.161 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-06-18T02:50:46.319 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install librados2 2024-06-18T02:50:46.444 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-06-18T02:50:46.626 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-06-18T02:50:46.681 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-06-18T02:50:46.886 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:01:34 ago on Tue 18 Jun 2024 02:49:12 AM UTC. 2024-06-18T02:50:46.971 INFO:teuthology.orchestra.run.smithi123.stdout:Package librados2-2:17.2.7-1012.g3fe41055.el9.x86_64 is already installed. 2024-06-18T02:50:47.013 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-06-18T02:50:47.014 INFO:teuthology.orchestra.run.smithi123.stdout:Nothing to do. 2024-06-18T02:50:47.014 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-06-18T02:50:47.076 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install librbd1 2024-06-18T02:50:47.245 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-06-18T02:50:47.335 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-06-18T02:50:47.398 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-06-18T02:50:47.425 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1012.g3fe41055.el9.noarch 6/6 2024-06-18T02:50:47.648 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:01:35 ago on Tue 18 Jun 2024 02:49:12 AM UTC. 2024-06-18T02:50:47.734 INFO:teuthology.orchestra.run.smithi123.stdout:Package librbd1-2:17.2.7-1012.g3fe41055.el9.x86_64 is already installed. 2024-06-18T02:50:47.775 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-06-18T02:50:47.776 INFO:teuthology.orchestra.run.smithi123.stdout:Nothing to do. 2024-06-18T02:50:47.777 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-06-18T02:50:47.836 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install python3-rados 2024-06-18T02:50:48.246 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1012.g3fe41055.el9.noarch 6/6 2024-06-18T02:50:48.246 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1012.g3fe41055.el9.noarch 1/6 2024-06-18T02:50:48.247 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-06-18T02:50:48.247 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-06-18T02:50:48.247 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-06-18T02:50:48.247 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-06-18T02:50:48.411 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:01:36 ago on Tue 18 Jun 2024 02:49:12 AM UTC. 2024-06-18T02:50:48.495 INFO:teuthology.orchestra.run.smithi123.stdout:Package python3-rados-2:17.2.7-1012.g3fe41055.el9.x86_64 is already installed. 2024-06-18T02:50:48.519 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-06-18T02:50:48.519 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:50:48.519 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-06-18T02:50:48.519 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-cephadm-2:17.2.7-1012.g3fe41055.el9.noarch 2024-06-18T02:50:48.519 INFO:teuthology.orchestra.run.smithi016.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-06-18T02:50:48.520 INFO:teuthology.orchestra.run.smithi016.stdout: python3-babel-2.9.1-2.el9.noarch 2024-06-18T02:50:48.520 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-06-18T02:50:48.520 INFO:teuthology.orchestra.run.smithi016.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-06-18T02:50:48.520 INFO:teuthology.orchestra.run.smithi016.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-06-18T02:50:48.520 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:50:48.520 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-06-18T02:50:48.537 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-06-18T02:50:48.538 INFO:teuthology.orchestra.run.smithi123.stdout:Nothing to do. 2024-06-18T02:50:48.538 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-06-18T02:50:48.602 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install python3-rgw 2024-06-18T02:50:48.761 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-fuse 2024-06-18T02:50:49.174 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:01:37 ago on Tue 18 Jun 2024 02:49:12 AM UTC. 2024-06-18T02:50:49.259 INFO:teuthology.orchestra.run.smithi123.stdout:Package python3-rgw-2:17.2.7-1012.g3fe41055.el9.x86_64 is already installed. 2024-06-18T02:50:49.301 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-06-18T02:50:49.302 INFO:teuthology.orchestra.run.smithi123.stdout:Nothing to do. 2024-06-18T02:50:49.302 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-06-18T02:50:49.359 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:39 ago on Tue 18 Jun 2024 02:49:10 AM UTC. 2024-06-18T02:50:49.370 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install python3-cephfs 2024-06-18T02:50:49.491 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-06-18T02:50:49.491 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-06-18T02:50:49.492 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2024-06-18T02:50:49.492 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-06-18T02:50:49.492 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-06-18T02:50:49.492 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-fuse x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 812 k 2024-06-18T02:50:49.492 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2024-06-18T02:50:49.492 INFO:teuthology.orchestra.run.smithi016.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-06-18T02:50:49.492 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:50:49.492 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-06-18T02:50:49.492 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-06-18T02:50:49.492 INFO:teuthology.orchestra.run.smithi016.stdout:Install 2 Packages 2024-06-18T02:50:49.492 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:50:49.493 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 892 k 2024-06-18T02:50:49.493 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 2.6 M 2024-06-18T02:50:49.493 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-06-18T02:50:49.730 INFO:teuthology.orchestra.run.smithi016.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 479 kB/s | 80 kB 00:00 2024-06-18T02:50:49.940 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:01:37 ago on Tue 18 Jun 2024 02:49:12 AM UTC. 2024-06-18T02:50:50.023 INFO:teuthology.orchestra.run.smithi123.stdout:Package python3-cephfs-2:17.2.7-1012.g3fe41055.el9.x86_64 is already installed. 2024-06-18T02:50:50.064 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-06-18T02:50:50.065 INFO:teuthology.orchestra.run.smithi123.stdout:Nothing to do. 2024-06-18T02:50:50.065 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-06-18T02:50:50.127 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install python3-rbd 2024-06-18T02:50:50.255 INFO:teuthology.orchestra.run.smithi016.stdout:(2/2): ceph-fuse-17.2.7-1012.g3fe41055.el9.x86_ 1.1 MB/s | 812 kB 00:00 2024-06-18T02:50:50.256 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-06-18T02:50:50.256 INFO:teuthology.orchestra.run.smithi016.stdout:Total 1.1 MB/s | 892 kB 00:00 2024-06-18T02:50:50.268 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-06-18T02:50:50.293 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-06-18T02:50:50.293 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-06-18T02:50:50.378 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-06-18T02:50:50.378 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-06-18T02:50:50.567 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-06-18T02:50:50.697 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-06-18T02:50:50.707 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:01:38 ago on Tue 18 Jun 2024 02:49:12 AM UTC. 2024-06-18T02:50:50.766 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-fuse-2:17.2.7-1012.g3fe41055.el9.x86_64 2/2 2024-06-18T02:50:50.792 INFO:teuthology.orchestra.run.smithi123.stdout:Package python3-rbd-2:17.2.7-1012.g3fe41055.el9.x86_64 is already installed. 2024-06-18T02:50:50.834 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-06-18T02:50:50.835 INFO:teuthology.orchestra.run.smithi123.stdout:Nothing to do. 2024-06-18T02:50:50.835 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-06-18T02:50:50.891 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install rbd-fuse 2024-06-18T02:50:51.415 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1012.g3fe41055.el9.x86_64 2/2 2024-06-18T02:50:51.416 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-fuse-2:17.2.7-1012.g3fe41055.el9.x86_64 1/2 2024-06-18T02:50:51.457 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:01:39 ago on Tue 18 Jun 2024 02:49:12 AM UTC. 2024-06-18T02:50:51.583 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-06-18T02:50:51.583 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-06-18T02:50:51.583 INFO:teuthology.orchestra.run.smithi123.stdout: Package Architecture Version Repository Size 2024-06-18T02:50:51.584 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-06-18T02:50:51.584 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-06-18T02:50:51.584 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-fuse x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 82 k 2024-06-18T02:50:51.584 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:50:51.584 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-06-18T02:50:51.584 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-06-18T02:50:51.584 INFO:teuthology.orchestra.run.smithi123.stdout:Install 1 Package 2024-06-18T02:50:51.584 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:50:51.584 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 82 k 2024-06-18T02:50:51.584 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 222 k 2024-06-18T02:50:51.584 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-06-18T02:50:51.661 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-06-18T02:50:51.661 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:50:51.661 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-06-18T02:50:51.661 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-fuse-2:17.2.7-1012.g3fe41055.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-06-18T02:50:51.661 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:50:51.661 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-06-18T02:50:51.874 INFO:teuthology.orchestra.run.smithi123.stdout:rbd-fuse-17.2.7-1012.g3fe41055.el9.x86_64.rpm 282 kB/s | 82 kB 00:00 2024-06-18T02:50:51.875 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-06-18T02:50:51.875 INFO:teuthology.orchestra.run.smithi123.stdout:Total 281 kB/s | 82 kB 00:00 2024-06-18T02:50:51.875 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-06-18T02:50:51.882 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-06-18T02:50:51.882 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-06-18T02:50:51.919 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install librados-devel 2024-06-18T02:50:51.943 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-06-18T02:50:51.944 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-06-18T02:50:52.088 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-06-18T02:50:52.179 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : rbd-fuse-2:17.2.7-1012.g3fe41055.el9.x86_64 1/1 2024-06-18T02:50:52.520 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:42 ago on Tue 18 Jun 2024 02:49:10 AM UTC. 2024-06-18T02:50:52.569 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1012.g3fe41055.el9.x86_64 1/1 2024-06-18T02:50:52.654 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-06-18T02:50:52.654 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-06-18T02:50:52.655 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repo Size 2024-06-18T02:50:52.655 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-06-18T02:50:52.655 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-06-18T02:50:52.655 INFO:teuthology.orchestra.run.smithi016.stdout: librados-devel x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 126 k 2024-06-18T02:50:52.655 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:50:52.655 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-06-18T02:50:52.655 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-06-18T02:50:52.655 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2024-06-18T02:50:52.655 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:50:52.655 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 126 k 2024-06-18T02:50:52.655 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 448 k 2024-06-18T02:50:52.655 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-06-18T02:50:52.816 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : rbd-fuse-2:17.2.7-1012.g3fe41055.el9.x86_64 1/1 2024-06-18T02:50:52.816 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:50:52.816 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-06-18T02:50:52.816 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-fuse-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:50:52.816 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:50:52.816 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-06-18T02:50:52.978 INFO:teuthology.orchestra.run.smithi016.stdout:librados-devel-17.2.7-1012.g3fe41055.el9.x86_64 392 kB/s | 126 kB 00:00 2024-06-18T02:50:52.978 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-06-18T02:50:52.978 INFO:teuthology.orchestra.run.smithi016.stdout:Total 390 kB/s | 126 kB 00:00 2024-06-18T02:50:52.979 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-06-18T02:50:52.985 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-06-18T02:50:52.985 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-06-18T02:50:53.033 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install rbd-mirror 2024-06-18T02:50:53.051 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-06-18T02:50:53.052 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-06-18T02:50:53.208 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-06-18T02:50:53.311 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librados-devel-2:17.2.7-1012.g3fe41055.el9.x86_64 1/1 2024-06-18T02:50:53.616 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:01:41 ago on Tue 18 Jun 2024 02:49:12 AM UTC. 2024-06-18T02:50:53.727 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librados-devel-2:17.2.7-1012.g3fe41055.el9.x86_64 1/1 2024-06-18T02:50:53.745 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-06-18T02:50:53.745 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-06-18T02:50:53.745 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repo Size 2024-06-18T02:50:53.745 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-06-18T02:50:53.746 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-06-18T02:50:53.746 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-mirror x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 2.9 M 2024-06-18T02:50:53.746 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:50:53.746 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-06-18T02:50:53.746 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-06-18T02:50:53.746 INFO:teuthology.orchestra.run.smithi123.stdout:Install 1 Package 2024-06-18T02:50:53.746 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:50:53.747 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 2.9 M 2024-06-18T02:50:53.747 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 12 M 2024-06-18T02:50:53.747 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-06-18T02:50:53.986 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librados-devel-2:17.2.7-1012.g3fe41055.el9.x86_64 1/1 2024-06-18T02:50:53.986 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:50:53.986 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-06-18T02:50:53.986 INFO:teuthology.orchestra.run.smithi016.stdout: librados-devel-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:50:53.986 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:50:53.986 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-06-18T02:50:54.202 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install libcephfs2 2024-06-18T02:50:54.801 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:44 ago on Tue 18 Jun 2024 02:49:10 AM UTC. 2024-06-18T02:50:54.835 INFO:teuthology.orchestra.run.smithi123.stdout:rbd-mirror-17.2.7-1012.g3fe41055.el9.x86_64.rpm 2.7 MB/s | 2.9 MB 00:01 2024-06-18T02:50:54.836 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-06-18T02:50:54.836 INFO:teuthology.orchestra.run.smithi123.stdout:Total 2.7 MB/s | 2.9 MB 00:01 2024-06-18T02:50:54.836 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-06-18T02:50:54.847 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-06-18T02:50:54.847 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-06-18T02:50:54.889 INFO:teuthology.orchestra.run.smithi016.stdout:Package libcephfs2-2:17.2.7-1012.g3fe41055.el9.x86_64 is already installed. 2024-06-18T02:50:54.929 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-06-18T02:50:54.929 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-06-18T02:50:54.933 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-06-18T02:50:54.934 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2024-06-18T02:50:54.935 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-06-18T02:50:54.998 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install libcephfs-devel 2024-06-18T02:50:55.351 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-06-18T02:50:55.419 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : rbd-mirror-2:17.2.7-1012.g3fe41055.el9.x86_64 1/1 2024-06-18T02:50:55.444 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1012.g3fe41055.el9.x86_64 1/1 2024-06-18T02:50:55.444 INFO:teuthology.orchestra.run.smithi123.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-18T02:50:55.444 INFO:teuthology.orchestra.run.smithi123.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-06-18T02:50:55.444 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-18T02:50:55.444 INFO:teuthology.orchestra.run.smithi123.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-18T02:50:55.444 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:50:55.597 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:45 ago on Tue 18 Jun 2024 02:49:10 AM UTC. 2024-06-18T02:50:55.728 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-06-18T02:50:55.729 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-06-18T02:50:55.729 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repo Size 2024-06-18T02:50:55.729 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-06-18T02:50:55.729 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-06-18T02:50:55.729 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs-devel x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 25 k 2024-06-18T02:50:55.729 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:50:55.729 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-06-18T02:50:55.729 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-06-18T02:50:55.730 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2024-06-18T02:50:55.730 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:50:55.730 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 25 k 2024-06-18T02:50:55.730 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 110 k 2024-06-18T02:50:55.730 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-06-18T02:50:55.916 INFO:teuthology.orchestra.run.smithi016.stdout:libcephfs-devel-17.2.7-1012.g3fe41055.el9.x86_6 134 kB/s | 25 kB 00:00 2024-06-18T02:50:55.916 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-06-18T02:50:55.917 INFO:teuthology.orchestra.run.smithi016.stdout:Total 133 kB/s | 25 kB 00:00 2024-06-18T02:50:55.917 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-06-18T02:50:55.920 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-06-18T02:50:55.920 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-06-18T02:50:55.922 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-06-18T02:50:55.922 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-06-18T02:50:56.000 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-06-18T02:50:56.124 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libcephfs-devel-2:17.2.7-1012.g3fe41055.el9.x86_64 1/1 2024-06-18T02:50:56.478 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : rbd-mirror-2:17.2.7-1012.g3fe41055.el9.x86_64 1/1 2024-06-18T02:50:56.478 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:50:56.478 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-06-18T02:50:56.478 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-mirror-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:50:56.478 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:50:56.478 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-06-18T02:50:56.490 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1012.g3fe41055.el9.x86_64 1/1 2024-06-18T02:50:56.714 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install rbd-nbd 2024-06-18T02:50:56.726 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libcephfs-devel-2:17.2.7-1012.g3fe41055.el9.x86_64 1/1 2024-06-18T02:50:56.726 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:50:56.726 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-06-18T02:50:56.726 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs-devel-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:50:56.726 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:50:56.726 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-06-18T02:50:56.898 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install librados2 2024-06-18T02:50:57.301 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:01:45 ago on Tue 18 Jun 2024 02:49:12 AM UTC. 2024-06-18T02:50:57.430 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2024-06-18T02:50:57.431 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-06-18T02:50:57.431 INFO:teuthology.orchestra.run.smithi123.stdout: Package Architecture Version Repository Size 2024-06-18T02:50:57.431 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-06-18T02:50:57.432 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2024-06-18T02:50:57.432 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-nbd x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 165 k 2024-06-18T02:50:57.432 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:50:57.432 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2024-06-18T02:50:57.432 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2024-06-18T02:50:57.432 INFO:teuthology.orchestra.run.smithi123.stdout:Install 1 Package 2024-06-18T02:50:57.432 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:50:57.432 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 165 k 2024-06-18T02:50:57.432 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 477 k 2024-06-18T02:50:57.432 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2024-06-18T02:50:57.510 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:47 ago on Tue 18 Jun 2024 02:49:10 AM UTC. 2024-06-18T02:50:57.600 INFO:teuthology.orchestra.run.smithi016.stdout:Package librados2-2:17.2.7-1012.g3fe41055.el9.x86_64 is already installed. 2024-06-18T02:50:57.645 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-06-18T02:50:57.646 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2024-06-18T02:50:57.646 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-06-18T02:50:57.707 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install librbd1 2024-06-18T02:50:57.771 INFO:teuthology.orchestra.run.smithi123.stdout:rbd-nbd-17.2.7-1012.g3fe41055.el9.x86_64.rpm 486 kB/s | 165 kB 00:00 2024-06-18T02:50:57.772 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2024-06-18T02:50:57.772 INFO:teuthology.orchestra.run.smithi123.stdout:Total 484 kB/s | 165 kB 00:00 2024-06-18T02:50:57.772 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2024-06-18T02:50:57.780 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2024-06-18T02:50:57.780 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2024-06-18T02:50:57.842 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2024-06-18T02:50:57.842 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2024-06-18T02:50:57.995 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2024-06-18T02:50:58.102 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : rbd-nbd-2:17.2.7-1012.g3fe41055.el9.x86_64 1/1 2024-06-18T02:50:58.312 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:48 ago on Tue 18 Jun 2024 02:49:10 AM UTC. 2024-06-18T02:50:58.400 INFO:teuthology.orchestra.run.smithi016.stdout:Package librbd1-2:17.2.7-1012.g3fe41055.el9.x86_64 is already installed. 2024-06-18T02:50:58.445 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-06-18T02:50:58.446 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2024-06-18T02:50:58.446 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-06-18T02:50:58.506 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install python3-rados 2024-06-18T02:50:58.544 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1012.g3fe41055.el9.x86_64 1/1 2024-06-18T02:50:58.785 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : rbd-nbd-2:17.2.7-1012.g3fe41055.el9.x86_64 1/1 2024-06-18T02:50:58.785 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:50:58.785 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2024-06-18T02:50:58.785 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-nbd-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:50:58.785 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:50:58.785 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2024-06-18T02:50:59.017 DEBUG:teuthology.parallel:result is None 2024-06-18T02:50:59.099 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:49 ago on Tue 18 Jun 2024 02:49:10 AM UTC. 2024-06-18T02:50:59.186 INFO:teuthology.orchestra.run.smithi016.stdout:Package python3-rados-2:17.2.7-1012.g3fe41055.el9.x86_64 is already installed. 2024-06-18T02:50:59.230 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-06-18T02:50:59.231 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2024-06-18T02:50:59.231 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-06-18T02:50:59.292 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install python3-rgw 2024-06-18T02:50:59.890 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:49 ago on Tue 18 Jun 2024 02:49:10 AM UTC. 2024-06-18T02:50:59.977 INFO:teuthology.orchestra.run.smithi016.stdout:Package python3-rgw-2:17.2.7-1012.g3fe41055.el9.x86_64 is already installed. 2024-06-18T02:51:00.021 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-06-18T02:51:00.022 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2024-06-18T02:51:00.022 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-06-18T02:51:00.083 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install python3-cephfs 2024-06-18T02:51:00.683 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:50 ago on Tue 18 Jun 2024 02:49:10 AM UTC. 2024-06-18T02:51:00.771 INFO:teuthology.orchestra.run.smithi016.stdout:Package python3-cephfs-2:17.2.7-1012.g3fe41055.el9.x86_64 is already installed. 2024-06-18T02:51:00.816 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-06-18T02:51:00.818 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2024-06-18T02:51:00.818 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-06-18T02:51:00.880 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install python3-rbd 2024-06-18T02:51:01.477 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:51 ago on Tue 18 Jun 2024 02:49:10 AM UTC. 2024-06-18T02:51:01.563 INFO:teuthology.orchestra.run.smithi016.stdout:Package python3-rbd-2:17.2.7-1012.g3fe41055.el9.x86_64 is already installed. 2024-06-18T02:51:01.607 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-06-18T02:51:01.608 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2024-06-18T02:51:01.609 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-06-18T02:51:01.671 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install rbd-fuse 2024-06-18T02:51:02.273 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:52 ago on Tue 18 Jun 2024 02:49:10 AM UTC. 2024-06-18T02:51:02.405 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-06-18T02:51:02.406 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-06-18T02:51:02.406 INFO:teuthology.orchestra.run.smithi016.stdout: Package Architecture Version Repository Size 2024-06-18T02:51:02.406 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-06-18T02:51:02.407 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-06-18T02:51:02.407 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-fuse x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 82 k 2024-06-18T02:51:02.407 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:51:02.407 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-06-18T02:51:02.407 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-06-18T02:51:02.407 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2024-06-18T02:51:02.407 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:51:02.407 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 82 k 2024-06-18T02:51:02.407 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 222 k 2024-06-18T02:51:02.407 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-06-18T02:51:02.620 INFO:teuthology.orchestra.run.smithi016.stdout:rbd-fuse-17.2.7-1012.g3fe41055.el9.x86_64.rpm 385 kB/s | 82 kB 00:00 2024-06-18T02:51:02.621 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-06-18T02:51:02.621 INFO:teuthology.orchestra.run.smithi016.stdout:Total 382 kB/s | 82 kB 00:00 2024-06-18T02:51:02.621 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-06-18T02:51:02.628 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-06-18T02:51:02.628 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-06-18T02:51:02.694 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-06-18T02:51:02.694 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-06-18T02:51:02.851 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-06-18T02:51:02.962 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : rbd-fuse-2:17.2.7-1012.g3fe41055.el9.x86_64 1/1 2024-06-18T02:51:03.347 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1012.g3fe41055.el9.x86_64 1/1 2024-06-18T02:51:03.619 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : rbd-fuse-2:17.2.7-1012.g3fe41055.el9.x86_64 1/1 2024-06-18T02:51:03.620 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:51:03.620 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-06-18T02:51:03.620 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-fuse-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:51:03.620 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:51:03.620 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-06-18T02:51:03.829 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install rbd-mirror 2024-06-18T02:51:04.429 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:54 ago on Tue 18 Jun 2024 02:49:10 AM UTC. 2024-06-18T02:51:04.563 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-06-18T02:51:04.563 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-06-18T02:51:04.563 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repo Size 2024-06-18T02:51:04.564 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-06-18T02:51:04.564 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-06-18T02:51:04.564 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-mirror x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 2.9 M 2024-06-18T02:51:04.564 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:51:04.564 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-06-18T02:51:04.564 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-06-18T02:51:04.564 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2024-06-18T02:51:04.564 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:51:04.564 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 2.9 M 2024-06-18T02:51:04.564 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 12 M 2024-06-18T02:51:04.564 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-06-18T02:51:05.157 INFO:teuthology.orchestra.run.smithi016.stdout:rbd-mirror-17.2.7-1012.g3fe41055.el9.x86_64.rpm 5.0 MB/s | 2.9 MB 00:00 2024-06-18T02:51:05.158 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-06-18T02:51:05.158 INFO:teuthology.orchestra.run.smithi016.stdout:Total 4.9 MB/s | 2.9 MB 00:00 2024-06-18T02:51:05.158 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-06-18T02:51:05.169 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-06-18T02:51:05.169 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-06-18T02:51:05.252 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-06-18T02:51:05.253 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-06-18T02:51:05.685 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-06-18T02:51:05.750 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : rbd-mirror-2:17.2.7-1012.g3fe41055.el9.x86_64 1/1 2024-06-18T02:51:05.776 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1012.g3fe41055.el9.x86_64 1/1 2024-06-18T02:51:05.777 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-18T02:51:05.777 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-06-18T02:51:05.777 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-18T02:51:05.777 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-18T02:51:05.777 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:51:06.976 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : rbd-mirror-2:17.2.7-1012.g3fe41055.el9.x86_64 1/1 2024-06-18T02:51:06.977 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:51:06.977 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-06-18T02:51:06.977 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-mirror-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:51:06.977 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:51:06.977 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-06-18T02:51:07.210 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install rbd-nbd 2024-06-18T02:51:07.807 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:01:57 ago on Tue 18 Jun 2024 02:49:10 AM UTC. 2024-06-18T02:51:07.958 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-06-18T02:51:07.959 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-06-18T02:51:07.959 INFO:teuthology.orchestra.run.smithi016.stdout: Package Architecture Version Repository Size 2024-06-18T02:51:07.959 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-06-18T02:51:07.959 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-06-18T02:51:07.959 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-nbd x86_64 2:17.2.7-1012.g3fe41055.el9 ceph 165 k 2024-06-18T02:51:07.959 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:51:07.959 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-06-18T02:51:07.959 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-06-18T02:51:07.959 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2024-06-18T02:51:07.959 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:51:07.960 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 165 k 2024-06-18T02:51:07.960 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 477 k 2024-06-18T02:51:07.960 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-06-18T02:51:08.203 INFO:teuthology.orchestra.run.smithi016.stdout:rbd-nbd-17.2.7-1012.g3fe41055.el9.x86_64.rpm 680 kB/s | 165 kB 00:00 2024-06-18T02:51:08.204 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-06-18T02:51:08.204 INFO:teuthology.orchestra.run.smithi016.stdout:Total 675 kB/s | 165 kB 00:00 2024-06-18T02:51:08.204 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-06-18T02:51:08.213 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-06-18T02:51:08.213 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-06-18T02:51:08.279 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-06-18T02:51:08.280 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-06-18T02:51:08.436 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-06-18T02:51:08.545 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : rbd-nbd-2:17.2.7-1012.g3fe41055.el9.x86_64 1/1 2024-06-18T02:51:08.954 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1012.g3fe41055.el9.x86_64 1/1 2024-06-18T02:51:09.208 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : rbd-nbd-2:17.2.7-1012.g3fe41055.el9.x86_64 1/1 2024-06-18T02:51:09.208 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:51:09.208 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-06-18T02:51:09.208 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-nbd-2:17.2.7-1012.g3fe41055.el9.x86_64 2024-06-18T02:51:09.208 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:51:09.208 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-06-18T02:51:09.450 DEBUG:teuthology.parallel:result is None 2024-06-18T02:51:09.451 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-18T02:51:09.451 INFO:teuthology.packaging:ref: None 2024-06-18T02:51:09.451 INFO:teuthology.packaging:tag: None 2024-06-18T02:51:09.451 INFO:teuthology.packaging:branch: quincy 2024-06-18T02:51:09.451 INFO:teuthology.packaging:sha1: 3781e3a592387d16e8020f5db99cdee4198f6d00 2024-06-18T02:51:09.451 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-06-18T02:51:09.833 DEBUG:teuthology.orchestra.run.smithi016:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-06-18T02:51:09.856 INFO:teuthology.orchestra.run.smithi016.stdout:17.2.7-1012.g3fe41055.el9 2024-06-18T02:51:09.857 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1012.g3fe41055.el9 2024-06-18T02:51:09.857 INFO:teuthology.task.install:The correct ceph version 17.2.7-1012.g3fe41055 is installed. 2024-06-18T02:51:09.859 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-18T02:51:09.859 INFO:teuthology.packaging:ref: None 2024-06-18T02:51:09.859 INFO:teuthology.packaging:tag: None 2024-06-18T02:51:09.859 INFO:teuthology.packaging:branch: quincy 2024-06-18T02:51:09.859 INFO:teuthology.packaging:sha1: 3781e3a592387d16e8020f5db99cdee4198f6d00 2024-06-18T02:51:09.859 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-06-18T02:51:10.196 DEBUG:teuthology.orchestra.run.smithi123:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-06-18T02:51:10.221 INFO:teuthology.orchestra.run.smithi123.stdout:17.2.7-1012.g3fe41055.el9 2024-06-18T02:51:10.222 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1012.g3fe41055.el9 2024-06-18T02:51:10.222 INFO:teuthology.task.install:The correct ceph version 17.2.7-1012.g3fe41055 is installed. 2024-06-18T02:51:10.224 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-06-18T02:51:10.224 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-06-18T02:51:10.224 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-06-18T02:51:10.254 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-06-18T02:51:10.254 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-06-18T02:51:10.290 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-06-18T02:51:10.290 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-06-18T02:51:10.290 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/usr/bin/daemon-helper 2024-06-18T02:51:10.323 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-06-18T02:51:10.390 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-06-18T02:51:10.391 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/usr/bin/daemon-helper 2024-06-18T02:51:10.417 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-06-18T02:51:10.483 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-06-18T02:51:10.484 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-06-18T02:51:10.484 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/usr/bin/adjust-ulimits 2024-06-18T02:51:10.511 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-06-18T02:51:10.577 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-06-18T02:51:10.577 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/usr/bin/adjust-ulimits 2024-06-18T02:51:10.604 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-06-18T02:51:10.670 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-06-18T02:51:10.670 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-06-18T02:51:10.670 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/usr/bin/stdin-killer 2024-06-18T02:51:10.697 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-06-18T02:51:10.763 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-06-18T02:51:10.763 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/usr/bin/stdin-killer 2024-06-18T02:51:10.788 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-06-18T02:51:10.854 INFO:teuthology.run_tasks:Running task print... 2024-06-18T02:51:10.866 INFO:teuthology.task.print:**** done install task... 2024-06-18T02:51:10.866 INFO:teuthology.run_tasks:Running task cephadm... 2024-06-18T02:51:10.953 INFO:tasks.cephadm:Config: {'cephadm_branch': 'quincy', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}}, 'image': 'quay.ceph.io/ceph-ci/ceph:quincy', 'roleless': True, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'OSD_DOWN', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy'], 'sha1': '3781e3a592387d16e8020f5db99cdee4198f6d00'} 2024-06-18T02:51:10.953 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:quincy 2024-06-18T02:51:10.953 INFO:tasks.cephadm:Cluster fsid is 9e93709a-2d1d-11ef-bc9d-c7b262605968 2024-06-18T02:51:10.954 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-06-18T02:51:10.954 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-06-18T02:51:10.954 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi016': '172.21.15.16', 'mon.smithi123': '172.21.15.123'} 2024-06-18T02:51:10.954 INFO:tasks.cephadm:Normalizing hostnames... 2024-06-18T02:51:10.954 DEBUG:teuthology.orchestra.run.smithi016:> sudo hostname $(hostname -s) 2024-06-18T02:51:10.981 DEBUG:teuthology.orchestra.run.smithi123:> sudo hostname $(hostname -s) 2024-06-18T02:51:11.006 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref quincy)... 2024-06-18T02:51:11.006 DEBUG:teuthology.orchestra.run.smithi016:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-06-18T02:51:11.303 INFO:teuthology.orchestra.run.smithi016.stdout:-rw-r--r--. 1 ubuntu ubuntu 375153 Jun 18 02:51 /home/ubuntu/cephtest/cephadm 2024-06-18T02:51:11.303 DEBUG:teuthology.orchestra.run.smithi123:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-06-18T02:51:11.505 INFO:teuthology.orchestra.run.smithi123.stdout:-rw-r--r--. 1 ubuntu ubuntu 375153 Jun 18 02:51 /home/ubuntu/cephtest/cephadm 2024-06-18T02:51:11.506 DEBUG:teuthology.orchestra.run.smithi016:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-06-18T02:51:11.526 DEBUG:teuthology.orchestra.run.smithi123:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-06-18T02:51:11.550 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:quincy on all hosts... 2024-06-18T02:51:11.550 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-06-18T02:51:11.568 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-06-18T02:51:12.450 INFO:teuthology.orchestra.run.smithi123.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-06-18T02:51:12.455 INFO:teuthology.orchestra.run.smithi016.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-06-18T02:51:38.717 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-06-18T02:51:38.717 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_version": "ceph version 17.2.7-1012-g3fe41055 (3fe410552b224c0fbe57dfbac3e18568c0f8fef6) quincy (stable)", 2024-06-18T02:51:38.717 INFO:teuthology.orchestra.run.smithi016.stdout: "image_id": "1a271607156e1b6becab587233eefa647399ff75aa2bb76fdb2f0360ec5b7818", 2024-06-18T02:51:38.717 INFO:teuthology.orchestra.run.smithi016.stdout: "repo_digests": [ 2024-06-18T02:51:38.717 INFO:teuthology.orchestra.run.smithi016.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032" 2024-06-18T02:51:38.718 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-06-18T02:51:38.718 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-06-18T02:51:45.409 INFO:teuthology.orchestra.run.smithi123.stdout:{ 2024-06-18T02:51:45.409 INFO:teuthology.orchestra.run.smithi123.stdout: "ceph_version": "ceph version 17.2.7-1012-g3fe41055 (3fe410552b224c0fbe57dfbac3e18568c0f8fef6) quincy (stable)", 2024-06-18T02:51:45.409 INFO:teuthology.orchestra.run.smithi123.stdout: "image_id": "1a271607156e1b6becab587233eefa647399ff75aa2bb76fdb2f0360ec5b7818", 2024-06-18T02:51:45.409 INFO:teuthology.orchestra.run.smithi123.stdout: "repo_digests": [ 2024-06-18T02:51:45.410 INFO:teuthology.orchestra.run.smithi123.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032" 2024-06-18T02:51:45.410 INFO:teuthology.orchestra.run.smithi123.stdout: ] 2024-06-18T02:51:45.410 INFO:teuthology.orchestra.run.smithi123.stdout:} 2024-06-18T02:51:45.429 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /etc/ceph 2024-06-18T02:51:45.465 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /etc/ceph 2024-06-18T02:51:45.493 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 777 /etc/ceph 2024-06-18T02:51:45.534 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 777 /etc/ceph 2024-06-18T02:51:45.559 INFO:tasks.cephadm:Writing seed config... 2024-06-18T02:51:45.560 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-06-18T02:51:45.560 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-06-18T02:51:45.560 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-06-18T02:51:45.560 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-06-18T02:51:45.560 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-06-18T02:51:45.561 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-06-18T02:51:45.561 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-06-18T02:51:45.561 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-06-18T02:51:45.561 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-06-18T02:51:45.561 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-06-18T02:51:45.561 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-06-18T02:51:45.561 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-06-18T02:51:45.561 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-06-18T02:51:45.561 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-06-18T02:51:45.561 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-06-18T02:51:45.561 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-06-18T02:51:45.561 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-06-18T02:51:45.561 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-06-18T02:51:45.561 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-06-18T02:51:45.561 INFO:tasks.cephadm: override: [client] debug client = 20 2024-06-18T02:51:45.562 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-06-18T02:51:45.562 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-06-18T02:51:45.562 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-06-18T02:51:45.562 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-06-18T02:51:45.562 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-06-18T02:51:45.562 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-06-18T02:51:45.562 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-06-18T02:51:45.562 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-06-18T02:51:45.562 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-06-18T02:51:45.562 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-06-18T02:51:45.562 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-06-18T02:51:45.562 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-06-18T02:51:45.562 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-06-18T02:51:45.562 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-06-18T02:51:45.563 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-06-18T02:51:45.563 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-06-18T02:51:45.563 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-06-18T02:51:45.563 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-06-18T02:51:45.563 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-06-18T02:51:45.563 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-06-18T02:51:45.563 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-06-18T02:51:45.563 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-06-18T02:51:45.564 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-06-18T02:51:45.564 DEBUG:teuthology.orchestra.run.smithi016:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-06-18T02:51:45.593 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 9e93709a-2d1d-11ef-bc9d-c7b262605968 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-06-18T02:51:45.593 DEBUG:teuthology.orchestra.run.smithi016:mon.smithi016> sudo journalctl -f -n 0 -u ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968@mon.smithi016.service 2024-06-18T02:51:45.635 INFO:tasks.cephadm:Bootstrapping... 2024-06-18T02:51:45.636 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy -v bootstrap --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.16 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-06-18T02:51:45.853 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-06-18T02:51:45.853 INFO:teuthology.orchestra.run.smithi016.stdout:cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:quincy', '-v', 'bootstrap', '--fsid', '9e93709a-2d1d-11ef-bc9d-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.16', '--skip-admin-label'] 2024-06-18T02:51:45.880 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stdout 5.1.0 2024-06-18T02:51:45.881 INFO:teuthology.orchestra.run.smithi016.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-06-18T02:51:45.881 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying podman|docker is present... 2024-06-18T02:51:45.906 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stdout 5.1.0 2024-06-18T02:51:45.906 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying lvm2 is present... 2024-06-18T02:51:45.906 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying time synchronization is in place... 2024-06-18T02:51:45.914 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-06-18T02:51:45.914 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-06-18T02:51:45.921 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-06-18T02:51:45.921 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout inactive 2024-06-18T02:51:45.928 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout enabled 2024-06-18T02:51:45.935 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout active 2024-06-18T02:51:45.935 INFO:teuthology.orchestra.run.smithi016.stdout:Unit chronyd.service is enabled and running 2024-06-18T02:51:45.935 INFO:teuthology.orchestra.run.smithi016.stdout:Repeating the final host check... 2024-06-18T02:51:45.961 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stdout 5.1.0 2024-06-18T02:51:45.961 INFO:teuthology.orchestra.run.smithi016.stdout:podman (/bin/podman) version 5.1.0 is present 2024-06-18T02:51:45.961 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl is present 2024-06-18T02:51:45.961 INFO:teuthology.orchestra.run.smithi016.stdout:lvcreate is present 2024-06-18T02:51:45.967 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-06-18T02:51:45.968 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-06-18T02:51:45.974 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-06-18T02:51:45.974 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout inactive 2024-06-18T02:51:45.981 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout enabled 2024-06-18T02:51:45.989 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout active 2024-06-18T02:51:45.989 INFO:teuthology.orchestra.run.smithi016.stdout:Unit chronyd.service is enabled and running 2024-06-18T02:51:45.989 INFO:teuthology.orchestra.run.smithi016.stdout:Host looks OK 2024-06-18T02:51:45.989 INFO:teuthology.orchestra.run.smithi016.stdout:Cluster fsid: 9e93709a-2d1d-11ef-bc9d-c7b262605968 2024-06-18T02:51:45.989 INFO:teuthology.orchestra.run.smithi016.stdout:Acquiring lock 139718389252544 on /run/cephadm/9e93709a-2d1d-11ef-bc9d-c7b262605968.lock 2024-06-18T02:51:45.989 INFO:teuthology.orchestra.run.smithi016.stdout:Lock 139718389252544 acquired on /run/cephadm/9e93709a-2d1d-11ef-bc9d-c7b262605968.lock 2024-06-18T02:51:45.990 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying IP 172.21.15.16 port 3300 ... 2024-06-18T02:51:45.990 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying IP 172.21.15.16 port 6789 ... 2024-06-18T02:51:45.990 INFO:teuthology.orchestra.run.smithi016.stdout:Base mon IP(s) is [172.21.15.16:3300, 172.21.15.16:6789], mon addrv is [v2:172.21.15.16:3300,v1:172.21.15.16:6789] 2024-06-18T02:51:45.993 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.16 metric 100 2024-06-18T02:51:45.993 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.16 metric 100 2024-06-18T02:51:45.996 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-06-18T02:51:45.996 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-06-18T02:51:45.997 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-06-18T02:51:45.999 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-06-18T02:51:45.999 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-06-18T02:51:45.999 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-06-18T02:51:45.999 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-06-18T02:51:45.999 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:1564/64 scope link noprefixroute 2024-06-18T02:51:46.000 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-06-18T02:51:46.000 INFO:teuthology.orchestra.run.smithi016.stdout:Mon IP `172.21.15.16` is in CIDR network `172.21.0.0/20` 2024-06-18T02:51:46.000 INFO:teuthology.orchestra.run.smithi016.stdout:Mon IP `172.21.15.16` is in CIDR network `172.21.0.0/20` 2024-06-18T02:51:46.000 INFO:teuthology.orchestra.run.smithi016.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-06-18T02:51:46.001 INFO:teuthology.orchestra.run.smithi016.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-06-18T02:51:46.001 INFO:teuthology.orchestra.run.smithi016.stdout:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-06-18T02:51:46.525 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stdout 1a271607156e1b6becab587233eefa647399ff75aa2bb76fdb2f0360ec5b7818 2024-06-18T02:51:46.525 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:quincy... 2024-06-18T02:51:46.525 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stderr Getting image source signatures 2024-06-18T02:51:46.525 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stderr Copying blob sha256:9065cd298533077f0006921bb12bd331d24f6c140cc8fa972c2e6c4e382038d1 2024-06-18T02:51:46.525 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stderr Copying blob sha256:7409f993fad97651003c683d751391d26d5a62a9a56c7e456e77f6286b65bea9 2024-06-18T02:51:46.525 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stderr Copying config sha256:1a271607156e1b6becab587233eefa647399ff75aa2bb76fdb2f0360ec5b7818 2024-06-18T02:51:46.525 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stderr Writing manifest to image destination 2024-06-18T02:51:47.414 INFO:teuthology.orchestra.run.smithi016.stdout:ceph: stdout ceph version 17.2.7-1012-g3fe41055 (3fe410552b224c0fbe57dfbac3e18568c0f8fef6) quincy (stable) 2024-06-18T02:51:47.414 INFO:teuthology.orchestra.run.smithi016.stdout:Ceph version: ceph version 17.2.7-1012-g3fe41055 (3fe410552b224c0fbe57dfbac3e18568c0f8fef6) quincy (stable) 2024-06-18T02:51:47.414 INFO:teuthology.orchestra.run.smithi016.stdout:Extracting ceph user uid/gid from container image... 2024-06-18T02:51:48.672 INFO:teuthology.orchestra.run.smithi016.stdout:stat: stdout 167 167 2024-06-18T02:51:48.672 INFO:teuthology.orchestra.run.smithi016.stdout:Creating initial keys... 2024-06-18T02:51:49.570 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph-authtool: stdout AQBE9nBmIZFSOhAA7RmWmP3HbdCMpp3RefQ5rQ== 2024-06-18T02:51:50.384 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph-authtool: stdout AQBF9nBmCGZCNBAAW9g6NoduoyvDfQliImH/oA== 2024-06-18T02:51:51.332 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph-authtool: stdout AQBG9nBmelwwKhAAvTgx4vi08AnpdFEp5N9foA== 2024-06-18T02:51:51.334 INFO:teuthology.orchestra.run.smithi016.stdout:Creating initial monmap... 2024-06-18T02:51:52.289 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-06-18T02:51:52.289 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = octopus 2024-06-18T02:51:52.289 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 9e93709a-2d1d-11ef-bc9d-c7b262605968 2024-06-18T02:51:52.289 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-06-18T02:51:52.289 INFO:teuthology.orchestra.run.smithi016.stdout:monmaptool for smithi016 [v2:172.21.15.16:3300,v1:172.21.15.16:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-06-18T02:51:52.289 INFO:teuthology.orchestra.run.smithi016.stdout:setting min_mon_release = octopus 2024-06-18T02:51:52.289 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/monmaptool: set fsid to 9e93709a-2d1d-11ef-bc9d-c7b262605968 2024-06-18T02:51:52.289 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-06-18T02:51:52.289 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:51:52.289 INFO:teuthology.orchestra.run.smithi016.stdout:Creating mon... 2024-06-18T02:51:53.228 INFO:teuthology.orchestra.run.smithi016.stdout:create mon.smithi016 on 2024-06-18T02:51:53.449 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-06-18T02:51:53.609 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-06-18T02:51:53.775 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968.target → /etc/systemd/system/ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968.target. 2024-06-18T02:51:53.775 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968.target → /etc/systemd/system/ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968.target. 2024-06-18T02:51:53.966 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968@mon.smithi016 2024-06-18T02:51:53.966 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Failed to reset failed state of unit ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968@mon.smithi016.service: Unit ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968@mon.smithi016.service not loaded. 2024-06-18T02:51:54.212 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968.target.wants/ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968@mon.smithi016.service → /etc/systemd/system/ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968@.service. 2024-06-18T02:51:54.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:51:54 smithi016 podman[28815]: 2024-06-18 02:51:54.374485489 +0000 UTC m=+0.018057241 image pull 1a271607156e1b6becab587233eefa647399ff75aa2bb76fdb2f0360ec5b7818 quay.ceph.io/ceph-ci/ceph:quincy 2024-06-18T02:51:55.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:51:55 smithi016 podman[28815]: 2024-06-18 02:51:55.476917834 +0000 UTC m=+1.120489579 container create 944141c920ce426f5ad1c921d41e05b0541225fe9796602e54db730dda014e8d (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-mon-smithi016, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240612, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=quincy-3fe4105, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True) 2024-06-18T02:51:57.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:51:57 smithi016 podman[28815]: 2024-06-18 02:51:57.27622996 +0000 UTC m=+2.919801720 container init 944141c920ce426f5ad1c921d41e05b0541225fe9796602e54db730dda014e8d (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-mon-smithi016, RELEASE=quincy-3fe4105, GIT_BRANCH=HEAD, org.label-schema.build-date=20240612, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.36.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True) 2024-06-18T02:51:57.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:51:57 smithi016 podman[28815]: 2024-06-18 02:51:57.280739872 +0000 UTC m=+2.924311619 container start 944141c920ce426f5ad1c921d41e05b0541225fe9796602e54db730dda014e8d (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-mon-smithi016, org.label-schema.build-date=20240612, RELEASE=quincy-3fe4105, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0) 2024-06-18T02:51:57.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:51:57 smithi016 ceph-mon[28835]: mkfs 9e93709a-2d1d-11ef-bc9d-c7b262605968 2024-06-18T02:51:57.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:51:57 smithi016 ceph-mon[28835]: mon.smithi016 is new leader, mons smithi016 in quorum (ranks 0) 2024-06-18T02:51:58.065 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-06-18T02:51:58.065 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout disabled 2024-06-18T02:51:58.074 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-06-18T02:51:58.075 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout inactive 2024-06-18T02:51:58.075 INFO:teuthology.orchestra.run.smithi016.stdout:firewalld.service is not enabled 2024-06-18T02:51:58.075 INFO:teuthology.orchestra.run.smithi016.stdout:Not possible to enable service . firewalld.service is not available 2024-06-18T02:51:58.075 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for mon to start... 2024-06-18T02:51:58.075 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for mon... 2024-06-18T02:51:58.305 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:51:58 smithi016 bash[28815]: 944141c920ce426f5ad1c921d41e05b0541225fe9796602e54db730dda014e8d 2024-06-18T02:51:58.305 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:51:58 smithi016 systemd[1]: Started Ceph mon.smithi016 for 9e93709a-2d1d-11ef-bc9d-c7b262605968. 2024-06-18T02:51:58.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:51:58 smithi016 ceph-mon[28835]: mon.smithi016 is new leader, mons smithi016 in quorum (ranks 0) 2024-06-18T02:51:58.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:51:58 smithi016 ceph-mon[28835]: monmap e1: 1 mons at {smithi016=[v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-06-18T02:51:58.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:51:58 smithi016 ceph-mon[28835]: fsmap 2024-06-18T02:51:58.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:51:58 smithi016 ceph-mon[28835]: osdmap e1: 0 total, 0 up, 0 in 2024-06-18T02:51:58.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:51:58 smithi016 ceph-mon[28835]: mgrmap e1: no daemons active 2024-06-18T02:52:01.305 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:01 smithi016 ceph-mon[28835]: from='client.? 172.21.15.16:0/2522719990' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-06-18T02:52:06.836 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout cluster: 2024-06-18T02:52:06.836 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout id: 9e93709a-2d1d-11ef-bc9d-c7b262605968 2024-06-18T02:52:06.836 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-06-18T02:52:06.836 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-06-18T02:52:06.836 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout services: 2024-06-18T02:52:06.836 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum smithi016 (age 3s) 2024-06-18T02:52:06.836 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-06-18T02:52:06.837 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-06-18T02:52:06.837 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-06-18T02:52:06.837 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout data: 2024-06-18T02:52:06.837 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-06-18T02:52:06.837 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-06-18T02:52:06.837 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-06-18T02:52:06.837 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout pgs: 2024-06-18T02:52:06.837 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-06-18T02:52:06.837 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:01.138+0000 7f1323453640 1 Processor -- start 2024-06-18T02:52:06.837 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:01.138+0000 7f1323453640 1 -- start start 2024-06-18T02:52:06.837 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:01.139+0000 7f1323453640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f131c101d90 0x7f131c1021a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:06.837 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:01.139+0000 7f1323453640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f131c1026e0 con 0x7f131c101d90 2024-06-18T02:52:06.837 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:01.139+0000 7f13211ef640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f131c101d90 0x7f131c1021a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:06.837 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:01.139+0000 7f13211ef640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f131c101d90 0x7f131c1021a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56636/0 (socket says 172.21.15.16:56636) 2024-06-18T02:52:06.838 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:01.139+0000 7f13211ef640 1 -- 172.21.15.16:0/961790606 learned_addr learned my addr 172.21.15.16:0/961790606 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:52:06.838 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:01.139+0000 7f13211ef640 1 -- 172.21.15.16:0/961790606 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f131c102820 con 0x7f131c101d90 2024-06-18T02:52:06.838 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:01.142+0000 7f13211ef640 1 --2- 172.21.15.16:0/961790606 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f131c101d90 0x7f131c1021a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f1318009840 tx=0x7f13180091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17ed0f553960901 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:06.838 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:01.142+0000 7f1313fff640 1 -- 172.21.15.16:0/961790606 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1318032440 con 0x7f131c101d90 2024-06-18T02:52:06.838 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:01.142+0000 7f1313fff640 1 -- 172.21.15.16:0/961790606 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13180325a0 con 0x7f131c101d90 2024-06-18T02:52:06.838 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:01.142+0000 7f1313fff640 1 -- 172.21.15.16:0/961790606 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f13180328d0 con 0x7f131c101d90 2024-06-18T02:52:06.838 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:01.142+0000 7f1323453640 1 -- 172.21.15.16:0/961790606 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f131c101d90 msgr2=0x7f131c1021a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:06.838 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:01.142+0000 7f1323453640 1 --2- 172.21.15.16:0/961790606 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f131c101d90 0x7f131c1021a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f1318009840 tx=0x7f13180091d0 comp rx=0 tx=0).stop 2024-06-18T02:52:06.838 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:01.143+0000 7f1323453640 1 -- 172.21.15.16:0/961790606 shutdown_connections 2024-06-18T02:52:06.838 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:01.143+0000 7f1323453640 1 --2- 172.21.15.16:0/961790606 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f131c101d90 0x7f131c1021a0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:06.838 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:01.143+0000 7f1323453640 1 -- 172.21.15.16:0/961790606 >> 172.21.15.16:0/961790606 conn(0x7f131c0fd5b0 msgr2=0x7f131c0ff9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:52:06.838 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:01.143+0000 7f1323453640 1 -- 172.21.15.16:0/961790606 shutdown_connections 2024-06-18T02:52:06.839 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:01.143+0000 7f1323453640 1 -- 172.21.15.16:0/961790606 wait complete. 2024-06-18T02:52:06.839 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:01.143+0000 7f1323453640 1 Processor -- start 2024-06-18T02:52:06.839 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:01.143+0000 7f1323453640 1 -- start start 2024-06-18T02:52:06.839 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:01.143+0000 7f1323453640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f131c101d90 0x7f131c110630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:06.839 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:01.143+0000 7f1323453640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f131c110b70 con 0x7f131c101d90 2024-06-18T02:52:06.839 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:01.144+0000 7f13211ef640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f131c101d90 0x7f131c110630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:06.839 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:01.144+0000 7f13211ef640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f131c101d90 0x7f131c110630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56638/0 (socket says 172.21.15.16:56638) 2024-06-18T02:52:06.839 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:01.144+0000 7f13211ef640 1 -- 172.21.15.16:0/2522719990 learned_addr learned my addr 172.21.15.16:0/2522719990 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:52:06.839 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:01.144+0000 7f13211ef640 1 -- 172.21.15.16:0/2522719990 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1318009450 con 0x7f131c101d90 2024-06-18T02:52:06.839 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:01.144+0000 7f13211ef640 1 --2- 172.21.15.16:0/2522719990 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f131c101d90 0x7f131c110630 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f131802a700 tx=0x7f1318032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:06.839 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:01.144+0000 7f1311ffb640 1 -- 172.21.15.16:0/2522719990 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1318031be0 con 0x7f131c101d90 2024-06-18T02:52:06.839 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:01.144+0000 7f1311ffb640 1 -- 172.21.15.16:0/2522719990 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1318031d40 con 0x7f131c101d90 2024-06-18T02:52:06.839 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:01.144+0000 7f1323453640 1 -- 172.21.15.16:0/2522719990 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f131c110d70 con 0x7f131c101d90 2024-06-18T02:52:06.839 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:01.144+0000 7f1311ffb640 1 -- 172.21.15.16:0/2522719990 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f131803b5a0 con 0x7f131c101d90 2024-06-18T02:52:06.840 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:01.145+0000 7f1323453640 1 -- 172.21.15.16:0/2522719990 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f131c111180 con 0x7f131c101d90 2024-06-18T02:52:06.840 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:01.145+0000 7f1311ffb640 1 -- 172.21.15.16:0/2522719990 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f1318039040 con 0x7f131c101d90 2024-06-18T02:52:06.840 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:01.145+0000 7f1311ffb640 1 -- 172.21.15.16:0/2522719990 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f1318041030 con 0x7f131c101d90 2024-06-18T02:52:06.840 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:01.146+0000 7f1323453640 1 -- 172.21.15.16:0/2522719990 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f131c04b480 con 0x7f131c101d90 2024-06-18T02:52:06.840 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:01.148+0000 7f1311ffb640 1 -- 172.21.15.16:0/2522719990 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71547 (secure 0 0 0) 0x7f1318037070 con 0x7f131c101d90 2024-06-18T02:52:06.840 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:01.213+0000 7f1323453640 1 -- 172.21.15.16:0/2522719990 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f131c103030 con 0x7f131c101d90 2024-06-18T02:52:06.840 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:01.213+0000 7f1311ffb640 1 -- 172.21.15.16:0/2522719990 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+318 (secure 0 0 0) 0x7f131803bbd0 con 0x7f131c101d90 2024-06-18T02:52:06.840 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:01.213+0000 7f1323453640 1 -- 172.21.15.16:0/2522719990 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f131c101d90 msgr2=0x7f131c110630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:06.840 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:01.213+0000 7f1323453640 1 --2- 172.21.15.16:0/2522719990 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f131c101d90 0x7f131c110630 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f131802a700 tx=0x7f1318032000 comp rx=0 tx=0).stop 2024-06-18T02:52:06.840 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:01.214+0000 7f1323453640 1 -- 172.21.15.16:0/2522719990 shutdown_connections 2024-06-18T02:52:06.840 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:01.214+0000 7f1323453640 1 --2- 172.21.15.16:0/2522719990 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f131c101d90 0x7f131c110630 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:06.840 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:01.214+0000 7f1323453640 1 -- 172.21.15.16:0/2522719990 >> 172.21.15.16:0/2522719990 conn(0x7f131c0fd5b0 msgr2=0x7f131c107220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:52:06.841 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:01.214+0000 7f1323453640 1 -- 172.21.15.16:0/2522719990 shutdown_connections 2024-06-18T02:52:06.841 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:01.214+0000 7f1323453640 1 -- 172.21.15.16:0/2522719990 wait complete. 2024-06-18T02:52:06.841 INFO:teuthology.orchestra.run.smithi016.stdout:mon is available 2024-06-18T02:52:06.841 INFO:teuthology.orchestra.run.smithi016.stdout:Assimilating anything we can from ceph.conf... 2024-06-18T02:52:07.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:07 smithi016 ceph-mon[28835]: from='client.? 172.21.15.16:0/2708413129' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-06-18T02:52:07.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:07 smithi016 ceph-mon[28835]: from='client.? 172.21.15.16:0/2708413129' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-06-18T02:52:07.936 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-06-18T02:52:07.936 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout [global] 2024-06-18T02:52:07.936 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout fsid = 9e93709a-2d1d-11ef-bc9d-c7b262605968 2024-06-18T02:52:07.936 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.16:3300,v1:172.21.15.16:6789] 2024-06-18T02:52:07.936 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-06-18T02:52:07.936 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-06-18T02:52:07.936 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-06-18T02:52:07.936 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-06-18T02:52:07.937 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-06-18T02:52:07.937 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout [mgr] 2024-06-18T02:52:07.937 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-06-18T02:52:07.937 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-06-18T02:52:07.937 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout [osd] 2024-06-18T02:52:07.937 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-06-18T02:52:07.937 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-06-18T02:52:07.937 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-06-18T02:52:07.937 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-06-18T02:52:07.937 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-06-18T02:52:07.937 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:07.243+0000 7f55f49c9640 1 Processor -- start 2024-06-18T02:52:07.937 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:07.244+0000 7f55f49c9640 1 -- start start 2024-06-18T02:52:07.937 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:07.244+0000 7f55f49c9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55f0075b40 0x7f55f0106da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:07.937 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:07.244+0000 7f55f49c9640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55f0075f50 con 0x7f55f0075b40 2024-06-18T02:52:07.938 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:07.244+0000 7f55ee59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55f0075b40 0x7f55f0106da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:07.938 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:07.244+0000 7f55ee59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55f0075b40 0x7f55f0106da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56652/0 (socket says 172.21.15.16:56652) 2024-06-18T02:52:07.938 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:07.244+0000 7f55ee59c640 1 -- 172.21.15.16:0/3255977113 learned_addr learned my addr 172.21.15.16:0/3255977113 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:52:07.938 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:07.244+0000 7f55ee59c640 1 -- 172.21.15.16:0/3255977113 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55f0102790 con 0x7f55f0075b40 2024-06-18T02:52:07.938 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:07.245+0000 7f55ee59c640 1 --2- 172.21.15.16:0/3255977113 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55f0075b40 0x7f55f0106da0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f55e0009840 tx=0x7f55e00091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cea8fca8363e800 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:07.938 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:07.245+0000 7f55edd9b640 1 -- 172.21.15.16:0/3255977113 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f55e0032440 con 0x7f55f0075b40 2024-06-18T02:52:07.938 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:07.245+0000 7f55edd9b640 1 -- 172.21.15.16:0/3255977113 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55e00325a0 con 0x7f55f0075b40 2024-06-18T02:52:07.938 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:07.245+0000 7f55edd9b640 1 -- 172.21.15.16:0/3255977113 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f55e00328d0 con 0x7f55f0075b40 2024-06-18T02:52:07.938 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:07.246+0000 7f55f49c9640 1 -- 172.21.15.16:0/3255977113 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55f0075b40 msgr2=0x7f55f0106da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:07.938 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:07.246+0000 7f55f49c9640 1 --2- 172.21.15.16:0/3255977113 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55f0075b40 0x7f55f0106da0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f55e0009840 tx=0x7f55e00091d0 comp rx=0 tx=0).stop 2024-06-18T02:52:07.938 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:07.246+0000 7f55f49c9640 1 -- 172.21.15.16:0/3255977113 shutdown_connections 2024-06-18T02:52:07.938 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:07.246+0000 7f55f49c9640 1 --2- 172.21.15.16:0/3255977113 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55f0075b40 0x7f55f0106da0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:07.938 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:07.246+0000 7f55f49c9640 1 -- 172.21.15.16:0/3255977113 >> 172.21.15.16:0/3255977113 conn(0x7f55f00fc430 msgr2=0x7f55f00fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:52:07.939 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:07.246+0000 7f55f49c9640 1 -- 172.21.15.16:0/3255977113 shutdown_connections 2024-06-18T02:52:07.939 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:07.246+0000 7f55f49c9640 1 -- 172.21.15.16:0/3255977113 wait complete. 2024-06-18T02:52:07.939 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:07.246+0000 7f55f49c9640 1 Processor -- start 2024-06-18T02:52:07.939 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:07.246+0000 7f55f49c9640 1 -- start start 2024-06-18T02:52:07.939 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:07.247+0000 7f55f49c9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55f0075b40 0x7f55f0112500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:07.939 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:07.247+0000 7f55f49c9640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55f0112a40 con 0x7f55f0075b40 2024-06-18T02:52:07.939 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:07.247+0000 7f55ee59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55f0075b40 0x7f55f0112500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:07.939 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:07.247+0000 7f55ee59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55f0075b40 0x7f55f0112500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56664/0 (socket says 172.21.15.16:56664) 2024-06-18T02:52:07.939 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:07.247+0000 7f55ee59c640 1 -- 172.21.15.16:0/2708413129 learned_addr learned my addr 172.21.15.16:0/2708413129 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:52:07.939 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:07.247+0000 7f55ee59c640 1 -- 172.21.15.16:0/2708413129 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55e0009450 con 0x7f55f0075b40 2024-06-18T02:52:07.939 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:07.247+0000 7f55ee59c640 1 --2- 172.21.15.16:0/2708413129 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55f0075b40 0x7f55f0112500 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f55e002a700 tx=0x7f55e0032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:07.939 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:07.247+0000 7f55e6ffd640 1 -- 172.21.15.16:0/2708413129 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f55e0031be0 con 0x7f55f0075b40 2024-06-18T02:52:07.939 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:07.247+0000 7f55e6ffd640 1 -- 172.21.15.16:0/2708413129 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55e0031d40 con 0x7f55f0075b40 2024-06-18T02:52:07.940 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:07.247+0000 7f55f49c9640 1 -- 172.21.15.16:0/2708413129 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55f0112c40 con 0x7f55f0075b40 2024-06-18T02:52:07.940 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:07.247+0000 7f55e6ffd640 1 -- 172.21.15.16:0/2708413129 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f55e003b5a0 con 0x7f55f0075b40 2024-06-18T02:52:07.940 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:07.247+0000 7f55f49c9640 1 -- 172.21.15.16:0/2708413129 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55f01137d0 con 0x7f55f0075b40 2024-06-18T02:52:07.940 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:07.248+0000 7f55e6ffd640 1 -- 172.21.15.16:0/2708413129 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f55e0039040 con 0x7f55f0075b40 2024-06-18T02:52:07.940 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:07.248+0000 7f55e6ffd640 1 -- 172.21.15.16:0/2708413129 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f55e0041070 con 0x7f55f0075b40 2024-06-18T02:52:07.940 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:07.248+0000 7f55f49c9640 1 -- 172.21.15.16:0/2708413129 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55f005ead0 con 0x7f55f0075b40 2024-06-18T02:52:07.940 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:07.251+0000 7f55e6ffd640 1 -- 172.21.15.16:0/2708413129 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71547 (secure 0 0 0) 0x7f55e0037070 con 0x7f55f0075b40 2024-06-18T02:52:07.940 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:07.312+0000 7f55f49c9640 1 -- 172.21.15.16:0/2708413129 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f55f0107590 con 0x7f55f0075b40 2024-06-18T02:52:07.940 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:07.315+0000 7f55e6ffd640 1 -- 172.21.15.16:0/2708413129 <== mon.0 v2:172.21.15.16:3300/0 7 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f55e004e8b0 con 0x7f55f0075b40 2024-06-18T02:52:07.940 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:07.315+0000 7f55e6ffd640 1 -- 172.21.15.16:0/2708413129 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7f55f0107590 con 0x7f55f0075b40 2024-06-18T02:52:07.940 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:07.317+0000 7f55f49c9640 1 -- 172.21.15.16:0/2708413129 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55f0075b40 msgr2=0x7f55f0112500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:07.940 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:07.317+0000 7f55f49c9640 1 --2- 172.21.15.16:0/2708413129 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55f0075b40 0x7f55f0112500 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f55e002a700 tx=0x7f55e0032000 comp rx=0 tx=0).stop 2024-06-18T02:52:07.940 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:07.317+0000 7f55f49c9640 1 -- 172.21.15.16:0/2708413129 shutdown_connections 2024-06-18T02:52:07.940 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:07.317+0000 7f55f49c9640 1 --2- 172.21.15.16:0/2708413129 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55f0075b40 0x7f55f0112500 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:07.941 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:07.317+0000 7f55f49c9640 1 -- 172.21.15.16:0/2708413129 >> 172.21.15.16:0/2708413129 conn(0x7f55f00fc430 msgr2=0x7f55f00fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:52:07.941 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:07.317+0000 7f55f49c9640 1 -- 172.21.15.16:0/2708413129 shutdown_connections 2024-06-18T02:52:07.941 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:07.317+0000 7f55f49c9640 1 -- 172.21.15.16:0/2708413129 wait complete. 2024-06-18T02:52:07.941 INFO:teuthology.orchestra.run.smithi016.stdout:Generating new minimal ceph.conf... 2024-06-18T02:52:08.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:08 smithi016 ceph-mon[28835]: from='client.? 172.21.15.16:0/2798747943' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:52:08.934 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:08.334+0000 7fda1b170640 1 Processor -- start 2024-06-18T02:52:08.934 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:08.334+0000 7fda1b170640 1 -- start start 2024-06-18T02:52:08.934 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:08.334+0000 7fda1b170640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda14075b00 0x7fda141049e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:08.935 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:08.334+0000 7fda1b170640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda14075f10 con 0x7fda14075b00 2024-06-18T02:52:08.935 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:08.334+0000 7fda18f0c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda14075b00 0x7fda141049e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:08.935 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:08.334+0000 7fda18f0c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda14075b00 0x7fda141049e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56668/0 (socket says 172.21.15.16:56668) 2024-06-18T02:52:08.935 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:08.334+0000 7fda18f0c640 1 -- 172.21.15.16:0/896748681 learned_addr learned my addr 172.21.15.16:0/896748681 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:52:08.935 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:08.334+0000 7fda18f0c640 1 -- 172.21.15.16:0/896748681 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda14104f20 con 0x7fda14075b00 2024-06-18T02:52:08.935 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:08.335+0000 7fda18f0c640 1 --2- 172.21.15.16:0/896748681 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda14075b00 0x7fda141049e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fda08009840 tx=0x7fda080091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ad93ad7cbe89c2a server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:08.935 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:08.335+0000 7fda137fe640 1 -- 172.21.15.16:0/896748681 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda08032440 con 0x7fda14075b00 2024-06-18T02:52:08.935 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:08.335+0000 7fda137fe640 1 -- 172.21.15.16:0/896748681 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fda08032a30 con 0x7fda14075b00 2024-06-18T02:52:08.935 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:08.335+0000 7fda137fe640 1 -- 172.21.15.16:0/896748681 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda08031ad0 con 0x7fda14075b00 2024-06-18T02:52:08.935 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:08.336+0000 7fda1b170640 1 -- 172.21.15.16:0/896748681 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda14075b00 msgr2=0x7fda141049e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:08.935 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:08.336+0000 7fda1b170640 1 --2- 172.21.15.16:0/896748681 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda14075b00 0x7fda141049e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fda08009840 tx=0x7fda080091d0 comp rx=0 tx=0).stop 2024-06-18T02:52:08.935 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:08.336+0000 7fda1b170640 1 -- 172.21.15.16:0/896748681 shutdown_connections 2024-06-18T02:52:08.936 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:08.336+0000 7fda1b170640 1 --2- 172.21.15.16:0/896748681 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda14075b00 0x7fda141049e0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:08.936 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:08.336+0000 7fda1b170640 1 -- 172.21.15.16:0/896748681 >> 172.21.15.16:0/896748681 conn(0x7fda140fc430 msgr2=0x7fda140fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:52:08.936 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:08.336+0000 7fda1b170640 1 -- 172.21.15.16:0/896748681 shutdown_connections 2024-06-18T02:52:08.936 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:08.336+0000 7fda1b170640 1 -- 172.21.15.16:0/896748681 wait complete. 2024-06-18T02:52:08.936 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:08.337+0000 7fda1b170640 1 Processor -- start 2024-06-18T02:52:08.936 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:08.337+0000 7fda1b170640 1 -- start start 2024-06-18T02:52:08.936 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:08.337+0000 7fda1b170640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda14075b00 0x7fda14102310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:08.936 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:08.337+0000 7fda1b170640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda14102850 con 0x7fda14075b00 2024-06-18T02:52:08.936 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:08.337+0000 7fda18f0c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda14075b00 0x7fda14102310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:08.936 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:08.337+0000 7fda18f0c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda14075b00 0x7fda14102310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56676/0 (socket says 172.21.15.16:56676) 2024-06-18T02:52:08.936 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:08.337+0000 7fda18f0c640 1 -- 172.21.15.16:0/2798747943 learned_addr learned my addr 172.21.15.16:0/2798747943 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:52:08.936 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:08.338+0000 7fda18f0c640 1 -- 172.21.15.16:0/2798747943 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda08009450 con 0x7fda14075b00 2024-06-18T02:52:08.936 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:08.338+0000 7fda18f0c640 1 --2- 172.21.15.16:0/2798747943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda14075b00 0x7fda14102310 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fda08006f00 tx=0x7fda08008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:08.936 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:08.338+0000 7fda117fa640 1 -- 172.21.15.16:0/2798747943 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda0803d450 con 0x7fda14075b00 2024-06-18T02:52:08.937 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:08.338+0000 7fda1b170640 1 -- 172.21.15.16:0/2798747943 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda14100940 con 0x7fda14075b00 2024-06-18T02:52:08.937 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:08.338+0000 7fda117fa640 1 -- 172.21.15.16:0/2798747943 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fda0803da40 con 0x7fda14075b00 2024-06-18T02:52:08.937 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:08.338+0000 7fda117fa640 1 -- 172.21.15.16:0/2798747943 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda0803bb10 con 0x7fda14075b00 2024-06-18T02:52:08.937 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:08.338+0000 7fda1b170640 1 -- 172.21.15.16:0/2798747943 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda14100e10 con 0x7fda14075b00 2024-06-18T02:52:08.937 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:08.338+0000 7fda117fa640 1 -- 172.21.15.16:0/2798747943 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7fda08039070 con 0x7fda14075b00 2024-06-18T02:52:08.937 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:08.338+0000 7fda117fa640 1 -- 172.21.15.16:0/2798747943 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fda08046930 con 0x7fda14075b00 2024-06-18T02:52:08.937 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:08.339+0000 7fda1b170640 1 -- 172.21.15.16:0/2798747943 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9f8005240 con 0x7fda14075b00 2024-06-18T02:52:08.937 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:08.341+0000 7fda117fa640 1 -- 172.21.15.16:0/2798747943 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71547 (secure 0 0 0) 0x7fda08041020 con 0x7fda14075b00 2024-06-18T02:52:08.937 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:08.403+0000 7fda1b170640 1 -- 172.21.15.16:0/2798747943 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fd9f80057c0 con 0x7fda14075b00 2024-06-18T02:52:08.937 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:08.403+0000 7fda117fa640 1 -- 172.21.15.16:0/2798747943 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7fda08037070 con 0x7fda14075b00 2024-06-18T02:52:08.937 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:08.404+0000 7fda1b170640 1 -- 172.21.15.16:0/2798747943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda14075b00 msgr2=0x7fda14102310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:08.937 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:08.404+0000 7fda1b170640 1 --2- 172.21.15.16:0/2798747943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda14075b00 0x7fda14102310 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fda08006f00 tx=0x7fda08008040 comp rx=0 tx=0).stop 2024-06-18T02:52:08.937 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:08.404+0000 7fda1b170640 1 -- 172.21.15.16:0/2798747943 shutdown_connections 2024-06-18T02:52:08.938 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:08.404+0000 7fda1b170640 1 --2- 172.21.15.16:0/2798747943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda14075b00 0x7fda14102310 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:08.938 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:08.404+0000 7fda1b170640 1 -- 172.21.15.16:0/2798747943 >> 172.21.15.16:0/2798747943 conn(0x7fda140fc430 msgr2=0x7fda140fcfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:52:08.938 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:08.404+0000 7fda1b170640 1 -- 172.21.15.16:0/2798747943 shutdown_connections 2024-06-18T02:52:08.938 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:08.404+0000 7fda1b170640 1 -- 172.21.15.16:0/2798747943 wait complete. 2024-06-18T02:52:08.938 INFO:teuthology.orchestra.run.smithi016.stdout:Restarting the monitor... 2024-06-18T02:52:09.214 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:08 smithi016 systemd[1]: Stopping Ceph mon.smithi016 for 9e93709a-2d1d-11ef-bc9d-c7b262605968... 2024-06-18T02:52:09.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:09 smithi016 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-mon-smithi016[28831]: 2024-06-18T02:52:09.213+0000 7f5f2bad2640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi016 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-06-18T02:52:09.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:09 smithi016 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-mon-smithi016[28831]: 2024-06-18T02:52:09.213+0000 7f5f2bad2640 -1 mon.smithi016@0(leader) e1 *** Got Signal Terminated *** 2024-06-18T02:52:09.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:09 smithi016 podman[29088]: 2024-06-18 02:52:09.37748628 +0000 UTC m=+0.271328078 container died 944141c920ce426f5ad1c921d41e05b0541225fe9796602e54db730dda014e8d (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-mon-smithi016, org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_BRANCH=HEAD, org.label-schema.build-date=20240612, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=quincy-3fe4105, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.36.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-06-18T02:52:09.897 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:09 smithi016 podman[29088]: 2024-06-18 02:52:09.654816942 +0000 UTC m=+0.548658739 container cleanup 944141c920ce426f5ad1c921d41e05b0541225fe9796602e54db730dda014e8d (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-mon-smithi016, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240612, RELEASE=quincy-3fe4105, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-06-18T02:52:09.897 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:09 smithi016 bash[29088]: ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-mon-smithi016 2024-06-18T02:52:10.239 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:09 smithi016 podman[29101]: 2024-06-18 02:52:09.897506116 +0000 UTC m=+0.517648890 container remove 944141c920ce426f5ad1c921d41e05b0541225fe9796602e54db730dda014e8d (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-mon-smithi016, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, RELEASE=quincy-3fe4105, org.label-schema.build-date=20240612, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-06-18T02:52:10.240 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 systemd[1]: ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968@mon.smithi016.service: Deactivated successfully. 2024-06-18T02:52:10.240 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 systemd[1]: Stopped Ceph mon.smithi016 for 9e93709a-2d1d-11ef-bc9d-c7b262605968. 2024-06-18T02:52:10.240 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 systemd[1]: Starting Ceph mon.smithi016 for 9e93709a-2d1d-11ef-bc9d-c7b262605968... 2024-06-18T02:52:10.497 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 podman[29195]: 2024-06-18 02:52:10.240100705 +0000 UTC m=+0.112074668 container create 01683559b4831b09c45624ab3e81d26706028b453ca350541850739dd740d737 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-mon-smithi016, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.build-date=20240612, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=quincy-3fe4105, org.label-schema.vendor=CentOS) 2024-06-18T02:52:10.497 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 podman[29195]: 2024-06-18 02:52:10.145303053 +0000 UTC m=+0.017277016 image pull 1a271607156e1b6becab587233eefa647399ff75aa2bb76fdb2f0360ec5b7818 quay.ceph.io/ceph-ci/ceph:quincy 2024-06-18T02:52:10.497 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 podman[29195]: 2024-06-18 02:52:10.476362356 +0000 UTC m=+0.348336319 container init 01683559b4831b09c45624ab3e81d26706028b453ca350541850739dd740d737 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-mon-smithi016, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240612, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-3fe4105) 2024-06-18T02:52:10.497 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 podman[29195]: 2024-06-18 02:52:10.479529231 +0000 UTC m=+0.351503202 container start 01683559b4831b09c45624ab3e81d26706028b453ca350541850739dd740d737 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-mon-smithi016, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240612, ceph=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_CLEAN=True, RELEASE=quincy-3fe4105, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-18T02:52:10.497 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: set uid:gid to 167:167 (ceph:ceph) 2024-06-18T02:52:10.498 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: ceph version 17.2.7-1012-g3fe41055 (3fe410552b224c0fbe57dfbac3e18568c0f8fef6) quincy (stable), process ceph-mon, pid 2 2024-06-18T02:52:10.498 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: pidfile_write: ignore empty --pid-file 2024-06-18T02:52:10.564 INFO:teuthology.orchestra.run.smithi016.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-06-18T02:52:10.787 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: load: jerasure load: lrc 2024-06-18T02:52:10.787 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: RocksDB version: 6.15.5 2024-06-18T02:52:10.787 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-06-18T02:52:10.787 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Compile date Jun 17 2024 2024-06-18T02:52:10.787 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: DB SUMMARY 2024-06-18T02:52:10.787 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: DB Session ID: LWPEV0K43FL2TUXUPYC4 2024-06-18T02:52:10.787 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: CURRENT file: CURRENT 2024-06-18T02:52:10.788 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: IDENTITY file: IDENTITY 2024-06-18T02:52:10.788 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-06-18T02:52:10.788 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi016/store.db dir, Total Num: 1, files: 000008.sst 2024-06-18T02:52:10.788 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi016/store.db: 000010.log size: 98470 ; 2024-06-18T02:52:10.788 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.error_if_exists: 0 2024-06-18T02:52:10.788 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.create_if_missing: 0 2024-06-18T02:52:10.788 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.paranoid_checks: 1 2024-06-18T02:52:10.788 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-06-18T02:52:10.789 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.env: 0x55d9872b3440 2024-06-18T02:52:10.789 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.fs: Posix File System 2024-06-18T02:52:10.789 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.info_log: 0x55d987dfb140 2024-06-18T02:52:10.789 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.max_file_opening_threads: 16 2024-06-18T02:52:10.789 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.statistics: (nil) 2024-06-18T02:52:10.789 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.use_fsync: 0 2024-06-18T02:52:10.789 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.max_log_file_size: 0 2024-06-18T02:52:10.789 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-06-18T02:52:10.789 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.log_file_time_to_roll: 0 2024-06-18T02:52:10.789 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.keep_log_file_num: 1000 2024-06-18T02:52:10.789 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.recycle_log_file_num: 0 2024-06-18T02:52:10.790 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.allow_fallocate: 1 2024-06-18T02:52:10.790 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.allow_mmap_reads: 0 2024-06-18T02:52:10.790 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.allow_mmap_writes: 0 2024-06-18T02:52:10.790 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.use_direct_reads: 0 2024-06-18T02:52:10.790 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-06-18T02:52:10.790 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.create_missing_column_families: 0 2024-06-18T02:52:10.790 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.db_log_dir: 2024-06-18T02:52:10.790 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi016/store.db 2024-06-18T02:52:10.790 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.table_cache_numshardbits: 6 2024-06-18T02:52:10.791 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.WAL_ttl_seconds: 0 2024-06-18T02:52:10.791 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.WAL_size_limit_MB: 0 2024-06-18T02:52:10.791 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-06-18T02:52:10.791 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-06-18T02:52:10.793 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.is_fd_close_on_exec: 1 2024-06-18T02:52:10.793 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.advise_random_on_open: 1 2024-06-18T02:52:10.793 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.db_write_buffer_size: 0 2024-06-18T02:52:10.793 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.write_buffer_manager: 0x55d987eb8840 2024-06-18T02:52:10.793 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-06-18T02:52:10.794 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-06-18T02:52:10.794 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-06-18T02:52:10.794 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.use_adaptive_mutex: 0 2024-06-18T02:52:10.794 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.rate_limiter: (nil) 2024-06-18T02:52:10.794 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-06-18T02:52:10.794 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.wal_recovery_mode: 2 2024-06-18T02:52:10.794 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.enable_thread_tracking: 0 2024-06-18T02:52:10.794 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.enable_pipelined_write: 0 2024-06-18T02:52:10.794 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.unordered_write: 0 2024-06-18T02:52:10.794 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-06-18T02:52:10.794 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-06-18T02:52:10.795 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-06-18T02:52:10.795 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-06-18T02:52:10.795 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.row_cache: None 2024-06-18T02:52:10.795 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.wal_filter: None 2024-06-18T02:52:10.795 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-06-18T02:52:10.795 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.allow_ingest_behind: 0 2024-06-18T02:52:10.795 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.preserve_deletes: 0 2024-06-18T02:52:10.795 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.two_write_queues: 0 2024-06-18T02:52:10.795 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.manual_wal_flush: 0 2024-06-18T02:52:10.795 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.atomic_flush: 0 2024-06-18T02:52:10.795 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-06-18T02:52:10.795 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.persist_stats_to_disk: 0 2024-06-18T02:52:10.796 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.write_dbid_to_manifest: 0 2024-06-18T02:52:10.796 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.log_readahead_size: 0 2024-06-18T02:52:10.796 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-06-18T02:52:10.796 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.best_efforts_recovery: 0 2024-06-18T02:52:10.796 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-06-18T02:52:10.796 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-06-18T02:52:10.796 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.allow_data_in_errors: 0 2024-06-18T02:52:10.796 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.db_host_id: __hostname__ 2024-06-18T02:52:10.796 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.max_background_jobs: 2 2024-06-18T02:52:10.796 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.max_background_compactions: -1 2024-06-18T02:52:10.796 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.max_subcompactions: 1 2024-06-18T02:52:10.796 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-06-18T02:52:10.796 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-06-18T02:52:10.796 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.delayed_write_rate : 16777216 2024-06-18T02:52:10.797 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.max_total_wal_size: 0 2024-06-18T02:52:10.797 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-06-18T02:52:10.797 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.stats_dump_period_sec: 600 2024-06-18T02:52:10.797 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.stats_persist_period_sec: 600 2024-06-18T02:52:10.797 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-06-18T02:52:10.797 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.max_open_files: -1 2024-06-18T02:52:10.797 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.bytes_per_sync: 0 2024-06-18T02:52:10.797 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.wal_bytes_per_sync: 0 2024-06-18T02:52:10.797 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.strict_bytes_per_sync: 0 2024-06-18T02:52:10.797 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.compaction_readahead_size: 0 2024-06-18T02:52:10.797 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.max_background_flushes: -1 2024-06-18T02:52:10.797 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Compression algorithms supported: 2024-06-18T02:52:10.797 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: kZSTD supported: 0 2024-06-18T02:52:10.798 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: kXpressCompression supported: 0 2024-06-18T02:52:10.798 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: kBZip2Compression supported: 0 2024-06-18T02:52:10.798 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-06-18T02:52:10.798 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: kLZ4Compression supported: 1 2024-06-18T02:52:10.798 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: kZlibCompression supported: 1 2024-06-18T02:52:10.798 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: kLZ4HCCompression supported: 1 2024-06-18T02:52:10.798 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: kSnappyCompression supported: 1 2024-06-18T02:52:10.798 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Fast CRC32 supported: Supported on x86 2024-06-18T02:52:10.798 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi016/store.db/MANIFEST-000009 2024-06-18T02:52:10.799 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-06-18T02:52:10.799 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-06-18T02:52:10.799 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.merge_operator: 2024-06-18T02:52:10.799 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.compaction_filter: None 2024-06-18T02:52:10.799 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.compaction_filter_factory: None 2024-06-18T02:52:10.799 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.sst_partitioner_factory: None 2024-06-18T02:52:10.799 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.memtable_factory: SkipListFactory 2024-06-18T02:52:10.799 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.table_factory: BlockBasedTable 2024-06-18T02:52:10.799 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55d987da1a68) 2024-06-18T02:52:10.799 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: cache_index_and_filter_blocks: 1 2024-06-18T02:52:10.799 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-06-18T02:52:10.800 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-06-18T02:52:10.800 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: pin_top_level_index_and_filter: 1 2024-06-18T02:52:10.800 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: index_type: 0 2024-06-18T02:52:10.800 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: data_block_index_type: 0 2024-06-18T02:52:10.800 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: index_shortening: 1 2024-06-18T02:52:10.800 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: data_block_hash_table_util_ratio: 0.750000 2024-06-18T02:52:10.800 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: hash_index_allow_collision: 1 2024-06-18T02:52:10.800 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: checksum: 1 2024-06-18T02:52:10.800 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: no_block_cache: 0 2024-06-18T02:52:10.800 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: block_cache: 0x55d987dd7350 2024-06-18T02:52:10.800 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: block_cache_name: BinnedLRUCache 2024-06-18T02:52:10.800 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: block_cache_options: 2024-06-18T02:52:10.800 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: capacity : 536870912 2024-06-18T02:52:10.800 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: num_shard_bits : 4 2024-06-18T02:52:10.801 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: strict_capacity_limit : 0 2024-06-18T02:52:10.801 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: high_pri_pool_ratio: 0.000 2024-06-18T02:52:10.801 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: block_cache_compressed: (nil) 2024-06-18T02:52:10.801 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: persistent_cache: (nil) 2024-06-18T02:52:10.801 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: block_size: 4096 2024-06-18T02:52:10.801 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: block_size_deviation: 10 2024-06-18T02:52:10.801 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: block_restart_interval: 16 2024-06-18T02:52:10.801 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: index_block_restart_interval: 1 2024-06-18T02:52:10.801 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: metadata_block_size: 4096 2024-06-18T02:52:10.801 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: partition_filters: 0 2024-06-18T02:52:10.801 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: use_delta_encoding: 1 2024-06-18T02:52:10.801 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-06-18T02:52:10.801 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: whole_key_filtering: 1 2024-06-18T02:52:10.801 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: verify_compression: 0 2024-06-18T02:52:10.802 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: read_amp_bytes_per_bit: 0 2024-06-18T02:52:10.802 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: format_version: 4 2024-06-18T02:52:10.802 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: enable_index_compression: 1 2024-06-18T02:52:10.802 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: block_align: 0 2024-06-18T02:52:10.802 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.write_buffer_size: 33554432 2024-06-18T02:52:10.802 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.max_write_buffer_number: 2 2024-06-18T02:52:10.802 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.compression: NoCompression 2024-06-18T02:52:10.802 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.bottommost_compression: Disabled 2024-06-18T02:52:10.802 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.prefix_extractor: nullptr 2024-06-18T02:52:10.802 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-06-18T02:52:10.802 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.num_levels: 7 2024-06-18T02:52:10.802 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-06-18T02:52:10.802 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-06-18T02:52:10.803 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-06-18T02:52:10.803 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-06-18T02:52:10.803 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-06-18T02:52:10.803 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-06-18T02:52:10.803 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-06-18T02:52:10.803 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-06-18T02:52:10.803 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-06-18T02:52:10.803 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-06-18T02:52:10.803 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.compression_opts.window_bits: -14 2024-06-18T02:52:10.803 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.compression_opts.level: 32767 2024-06-18T02:52:10.803 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.compression_opts.strategy: 0 2024-06-18T02:52:10.803 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-06-18T02:52:10.803 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-06-18T02:52:10.803 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-06-18T02:52:10.804 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.compression_opts.enabled: false 2024-06-18T02:52:10.804 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-06-18T02:52:10.804 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-06-18T02:52:10.804 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-06-18T02:52:10.804 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.target_file_size_base: 67108864 2024-06-18T02:52:10.804 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.target_file_size_multiplier: 1 2024-06-18T02:52:10.804 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-06-18T02:52:10.804 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-06-18T02:52:10.804 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-06-18T02:52:10.804 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-06-18T02:52:10.804 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-06-18T02:52:10.804 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-06-18T02:52:10.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-06-18T02:52:10.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-06-18T02:52:10.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-06-18T02:52:10.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-06-18T02:52:10.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-06-18T02:52:10.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-06-18T02:52:10.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.arena_block_size: 4194304 2024-06-18T02:52:10.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-06-18T02:52:10.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-06-18T02:52:10.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-06-18T02:52:10.806 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.disable_auto_compactions: 0 2024-06-18T02:52:10.806 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-06-18T02:52:10.806 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-06-18T02:52:10.806 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-06-18T02:52:10.806 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-06-18T02:52:10.806 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-06-18T02:52:10.806 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-06-18T02:52:10.806 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-06-18T02:52:10.806 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-06-18T02:52:10.806 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-06-18T02:52:10.806 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-06-18T02:52:10.806 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.table_properties_collectors: 2024-06-18T02:52:10.807 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.inplace_update_support: 0 2024-06-18T02:52:10.807 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.inplace_update_num_locks: 10000 2024-06-18T02:52:10.807 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-06-18T02:52:10.807 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-06-18T02:52:10.807 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.memtable_huge_page_size: 0 2024-06-18T02:52:10.807 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.bloom_locality: 0 2024-06-18T02:52:10.807 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.max_successive_merges: 0 2024-06-18T02:52:10.807 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.optimize_filters_for_hits: 0 2024-06-18T02:52:10.807 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.paranoid_file_checks: 0 2024-06-18T02:52:10.807 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.force_consistency_checks: 1 2024-06-18T02:52:10.808 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.report_bg_io_stats: 0 2024-06-18T02:52:10.808 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.ttl: 2592000 2024-06-18T02:52:10.808 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.periodic_compaction_seconds: 0 2024-06-18T02:52:10.808 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.enable_blob_files: false 2024-06-18T02:52:10.808 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.min_blob_size: 0 2024-06-18T02:52:10.808 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.blob_file_size: 268435456 2024-06-18T02:52:10.808 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.blob_compression_type: NoCompression 2024-06-18T02:52:10.808 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.enable_blob_garbage_collection: false 2024-06-18T02:52:10.808 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-06-18T02:52:10.808 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi016/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-06-18T02:52:10.809 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 5 2024-06-18T02:52:10.809 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: [db/version_set.cc:4082] Creating manifest 13 2024-06-18T02:52:10.809 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: EVENT_LOG_v1 {"time_micros": 1718679130502409, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-06-18T02:52:10.809 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #10 mode 2 2024-06-18T02:52:10.809 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-06-18T02:52:10.809 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: EVENT_LOG_v1 {"time_micros": 1718679130503421, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 93521, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 91633, "index_size": 248, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13606, "raw_average_key_size": 49, "raw_value_size": 84171, "raw_average_value_size": 306, "num_data_blocks": 11, "num_entries": 275, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1718679130, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "9bc2dbc7-e1a7-4576-9b3d-cfc86f96aa02", "db_session_id": "LWPEV0K43FL2TUXUPYC4"}} 2024-06-18T02:52:10.810 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: [db/version_set.cc:4082] Creating manifest 15 2024-06-18T02:52:10.810 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: EVENT_LOG_v1 {"time_micros": 1718679130503773, "job": 1, "event": "recovery_finished"} 2024-06-18T02:52:10.810 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-smithi016/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-06-18T02:52:10.810 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x55d987e0a700 2024-06-18T02:52:10.810 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: DB pointer 0x55d987dda000 2024-06-18T02:52:10.810 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-06-18T02:52:10.811 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-06-18T02:52:10.811 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: ** DB Stats ** 2024-06-18T02:52:10.811 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-18T02:52:10.811 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-06-18T02:52:10.811 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-06-18T02:52:10.811 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-06-18T02:52:10.812 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-06-18T02:52:10.812 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-06-18T02:52:10.812 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-06-18T02:52:10.812 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: 2024-06-18T02:52:10.812 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: ** Compaction Stats [default] ** 2024-06-18T02:52:10.812 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-06-18T02:52:10.812 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-06-18T02:52:10.813 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: L0 2/0 93.13 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 126.0 0.00 0.00 1 0.001 0 0 2024-06-18T02:52:10.813 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: Sum 2/0 93.13 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 126.0 0.00 0.00 1 0.001 0 0 2024-06-18T02:52:10.813 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 126.0 0.00 0.00 1 0.001 0 0 2024-06-18T02:52:10.813 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: 2024-06-18T02:52:10.813 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: ** Compaction Stats [default] ** 2024-06-18T02:52:10.813 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-06-18T02:52:10.813 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-06-18T02:52:10.814 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 126.0 0.00 0.00 1 0.001 0 0 2024-06-18T02:52:10.814 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-18T02:52:10.814 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-06-18T02:52:10.814 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-06-18T02:52:10.814 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-06-18T02:52:10.814 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-06-18T02:52:10.814 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: AddFile(Keys): cumulative 0, interval 0 2024-06-18T02:52:10.814 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: Cumulative compaction: 0.00 GB write, 26.52 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-18T02:52:10.815 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: Interval compaction: 0.00 GB write, 26.52 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-18T02:52:10.815 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-06-18T02:52:10.815 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: 2024-06-18T02:52:10.815 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: ** File Read Latency Histogram By Level [default] ** 2024-06-18T02:52:10.815 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: 2024-06-18T02:52:10.815 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: ** Compaction Stats [default] ** 2024-06-18T02:52:10.815 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-06-18T02:52:10.815 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-06-18T02:52:10.815 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: L0 2/0 93.13 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 126.0 0.00 0.00 1 0.001 0 0 2024-06-18T02:52:10.815 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: Sum 2/0 93.13 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 126.0 0.00 0.00 1 0.001 0 0 2024-06-18T02:52:10.816 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-06-18T02:52:10.816 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: 2024-06-18T02:52:10.816 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: ** Compaction Stats [default] ** 2024-06-18T02:52:10.816 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-06-18T02:52:10.816 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-06-18T02:52:10.816 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 126.0 0.00 0.00 1 0.001 0 0 2024-06-18T02:52:10.816 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-18T02:52:10.816 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-06-18T02:52:10.816 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-06-18T02:52:10.816 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-06-18T02:52:10.816 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-06-18T02:52:10.817 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: AddFile(Keys): cumulative 0, interval 0 2024-06-18T02:52:10.817 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: Cumulative compaction: 0.00 GB write, 26.27 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-18T02:52:10.817 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-18T02:52:10.817 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-06-18T02:52:10.817 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: 2024-06-18T02:52:10.817 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: ** File Read Latency Histogram By Level [default] ** 2024-06-18T02:52:10.817 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: starting mon.smithi016 rank 0 at public addrs [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] at bind addrs [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi016 fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 2024-06-18T02:52:10.817 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: mon.smithi016@-1(???) e1 preinit fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 2024-06-18T02:52:10.817 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: mon.smithi016@-1(???).mds e1 new map 2024-06-18T02:52:10.818 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: mon.smithi016@-1(???).mds e1 print_map 2024-06-18T02:52:10.818 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: e1 2024-06-18T02:52:10.818 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-06-18T02:52:10.818 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-06-18T02:52:10.818 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: legacy client fscid: -1 2024-06-18T02:52:10.819 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: 2024-06-18T02:52:10.819 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: No filesystems configured 2024-06-18T02:52:10.819 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: mon.smithi016@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-06-18T02:52:10.819 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: mon.smithi016@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-06-18T02:52:10.820 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: mon.smithi016@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-06-18T02:52:10.820 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: mon.smithi016@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-06-18T02:52:10.820 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: mon.smithi016@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-06-18T02:52:10.821 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 bash[29195]: 01683559b4831b09c45624ab3e81d26706028b453ca350541850739dd740d737 2024-06-18T02:52:10.821 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: mon.smithi016 is new leader, mons smithi016 in quorum (ranks 0) 2024-06-18T02:52:10.821 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: monmap e1: 1 mons at {smithi016=[v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-06-18T02:52:10.822 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: fsmap 2024-06-18T02:52:10.822 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: osdmap e1: 0 total, 0 up, 0 in 2024-06-18T02:52:10.822 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 ceph-mon[29214]: mgrmap e1: no daemons active 2024-06-18T02:52:10.822 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:10 smithi016 systemd[1]: Started Ceph mon.smithi016 for 9e93709a-2d1d-11ef-bc9d-c7b262605968. 2024-06-18T02:52:11.581 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:10.977+0000 7f9c688f9640 1 Processor -- start 2024-06-18T02:52:11.582 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:10.977+0000 7f9c688f9640 1 -- start start 2024-06-18T02:52:11.582 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:10.978+0000 7f9c688f9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c641000d0 0x7f9c641004e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:11.582 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:10.978+0000 7f9c688f9640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c64100a20 con 0x7f9c641000d0 2024-06-18T02:52:11.582 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:10.978+0000 7f9c6259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c641000d0 0x7f9c641004e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:11.582 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:10.978+0000 7f9c6259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c641000d0 0x7f9c641004e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45736/0 (socket says 172.21.15.16:45736) 2024-06-18T02:52:11.582 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:10.978+0000 7f9c6259c640 1 -- 172.21.15.16:0/2597312703 learned_addr learned my addr 172.21.15.16:0/2597312703 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:52:11.582 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:10.978+0000 7f9c6259c640 1 -- 172.21.15.16:0/2597312703 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c64100b60 con 0x7f9c641000d0 2024-06-18T02:52:11.582 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:10.981+0000 7f9c6259c640 1 --2- 172.21.15.16:0/2597312703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c641000d0 0x7f9c641004e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f9c54009840 tx=0x7f9c540091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8df0aee33113d8c server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:11.582 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:10.981+0000 7f9c61d9b640 1 -- 172.21.15.16:0/2597312703 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c54032440 con 0x7f9c641000d0 2024-06-18T02:52:11.582 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:10.981+0000 7f9c61d9b640 1 -- 172.21.15.16:0/2597312703 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f9c54032a30 con 0x7f9c641000d0 2024-06-18T02:52:11.582 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:10.981+0000 7f9c61d9b640 1 -- 172.21.15.16:0/2597312703 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c54031ad0 con 0x7f9c641000d0 2024-06-18T02:52:11.583 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:10.981+0000 7f9c688f9640 1 -- 172.21.15.16:0/2597312703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c641000d0 msgr2=0x7f9c641004e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:11.583 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:10.981+0000 7f9c688f9640 1 --2- 172.21.15.16:0/2597312703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c641000d0 0x7f9c641004e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f9c54009840 tx=0x7f9c540091d0 comp rx=0 tx=0).stop 2024-06-18T02:52:11.583 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:10.982+0000 7f9c688f9640 1 -- 172.21.15.16:0/2597312703 shutdown_connections 2024-06-18T02:52:11.583 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:10.982+0000 7f9c688f9640 1 --2- 172.21.15.16:0/2597312703 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c641000d0 0x7f9c641004e0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:11.583 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:10.982+0000 7f9c688f9640 1 -- 172.21.15.16:0/2597312703 >> 172.21.15.16:0/2597312703 conn(0x7f9c640fbc50 msgr2=0x7f9c640fe060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:52:11.583 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:10.982+0000 7f9c688f9640 1 -- 172.21.15.16:0/2597312703 shutdown_connections 2024-06-18T02:52:11.583 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:10.982+0000 7f9c688f9640 1 -- 172.21.15.16:0/2597312703 wait complete. 2024-06-18T02:52:11.583 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:10.982+0000 7f9c688f9640 1 Processor -- start 2024-06-18T02:52:11.583 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:10.982+0000 7f9c688f9640 1 -- start start 2024-06-18T02:52:11.583 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:10.982+0000 7f9c688f9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c641000d0 0x7f9c641170e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:11.583 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:10.982+0000 7f9c688f9640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c64117620 con 0x7f9c641000d0 2024-06-18T02:52:11.583 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:10.982+0000 7f9c6259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c641000d0 0x7f9c641170e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:11.583 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:10.983+0000 7f9c6259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c641000d0 0x7f9c641170e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45748/0 (socket says 172.21.15.16:45748) 2024-06-18T02:52:11.583 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:10.983+0000 7f9c6259c640 1 -- 172.21.15.16:0/457614034 learned_addr learned my addr 172.21.15.16:0/457614034 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:52:11.584 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:10.983+0000 7f9c6259c640 1 -- 172.21.15.16:0/457614034 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c54009450 con 0x7f9c641000d0 2024-06-18T02:52:11.584 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:10.983+0000 7f9c6259c640 1 --2- 172.21.15.16:0/457614034 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c641000d0 0x7f9c641170e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f9c54031b90 tx=0x7f9c54008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:11.584 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:10.983+0000 7f9c5affd640 1 -- 172.21.15.16:0/457614034 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c540328b0 con 0x7f9c641000d0 2024-06-18T02:52:11.584 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:10.983+0000 7f9c5affd640 1 -- 172.21.15.16:0/457614034 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f9c5403d8e0 con 0x7f9c641000d0 2024-06-18T02:52:11.584 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:10.983+0000 7f9c5affd640 1 -- 172.21.15.16:0/457614034 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c5403ba20 con 0x7f9c641000d0 2024-06-18T02:52:11.584 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:10.983+0000 7f9c688f9640 1 -- 172.21.15.16:0/457614034 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c64117820 con 0x7f9c641000d0 2024-06-18T02:52:11.584 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:10.983+0000 7f9c688f9640 1 -- 172.21.15.16:0/457614034 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c64117c90 con 0x7f9c641000d0 2024-06-18T02:52:11.584 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:10.983+0000 7f9c5affd640 1 -- 172.21.15.16:0/457614034 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f9c54039070 con 0x7f9c641000d0 2024-06-18T02:52:11.584 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:10.983+0000 7f9c5affd640 1 -- 172.21.15.16:0/457614034 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f9c54046930 con 0x7f9c641000d0 2024-06-18T02:52:11.584 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:10.984+0000 7f9c688f9640 1 -- 172.21.15.16:0/457614034 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c6404b480 con 0x7f9c641000d0 2024-06-18T02:52:11.584 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:10.986+0000 7f9c5affd640 1 -- 172.21.15.16:0/457614034 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71547 (secure 0 0 0) 0x7f9c54065790 con 0x7f9c641000d0 2024-06-18T02:52:11.584 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:11.047+0000 7f9c688f9640 1 -- 172.21.15.16:0/457614034 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f9c64101970 con 0x7f9c641000d0 2024-06-18T02:52:11.584 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:11.049+0000 7f9c5affd640 1 -- 172.21.15.16:0/457614034 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f9c54054020 con 0x7f9c641000d0 2024-06-18T02:52:11.584 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:11.050+0000 7f9c688f9640 1 -- 172.21.15.16:0/457614034 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c641000d0 msgr2=0x7f9c641170e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:11.585 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:11.050+0000 7f9c688f9640 1 --2- 172.21.15.16:0/457614034 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c641000d0 0x7f9c641170e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f9c54031b90 tx=0x7f9c54008040 comp rx=0 tx=0).stop 2024-06-18T02:52:11.585 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:11.050+0000 7f9c688f9640 1 -- 172.21.15.16:0/457614034 shutdown_connections 2024-06-18T02:52:11.585 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:11.050+0000 7f9c688f9640 1 --2- 172.21.15.16:0/457614034 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c641000d0 0x7f9c641170e0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:11.585 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:11.050+0000 7f9c688f9640 1 -- 172.21.15.16:0/457614034 >> 172.21.15.16:0/457614034 conn(0x7f9c640fbc50 msgr2=0x7f9c640fc7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:52:11.585 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:11.050+0000 7f9c688f9640 1 -- 172.21.15.16:0/457614034 shutdown_connections 2024-06-18T02:52:11.585 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:11.050+0000 7f9c688f9640 1 -- 172.21.15.16:0/457614034 wait complete. 2024-06-18T02:52:11.585 INFO:teuthology.orchestra.run.smithi016.stdout:Wrote config to /etc/ceph/ceph.conf 2024-06-18T02:52:11.585 INFO:teuthology.orchestra.run.smithi016.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-06-18T02:52:11.585 INFO:teuthology.orchestra.run.smithi016.stdout:Creating mgr... 2024-06-18T02:52:11.585 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying port 9283 ... 2024-06-18T02:52:11.763 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968@mgr.smithi016.amxydz 2024-06-18T02:52:11.764 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Failed to reset failed state of unit ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968@mgr.smithi016.amxydz.service: Unit ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968@mgr.smithi016.amxydz.service not loaded. 2024-06-18T02:52:11.919 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968.target.wants/ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968@mgr.smithi016.amxydz.service → /etc/systemd/system/ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968@.service. 2024-06-18T02:52:12.305 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:12 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/457614034' entity='client.admin' 2024-06-18T02:52:12.926 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-06-18T02:52:12.926 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout disabled 2024-06-18T02:52:12.935 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-06-18T02:52:12.935 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout inactive 2024-06-18T02:52:12.935 INFO:teuthology.orchestra.run.smithi016.stdout:firewalld.service is not enabled 2024-06-18T02:52:12.935 INFO:teuthology.orchestra.run.smithi016.stdout:Not possible to enable service . firewalld.service is not available 2024-06-18T02:52:12.945 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-06-18T02:52:12.945 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout disabled 2024-06-18T02:52:12.954 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-06-18T02:52:12.954 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout inactive 2024-06-18T02:52:12.954 INFO:teuthology.orchestra.run.smithi016.stdout:firewalld.service is not enabled 2024-06-18T02:52:12.954 INFO:teuthology.orchestra.run.smithi016.stdout:Not possible to open ports <[9283]>. firewalld.service is not available 2024-06-18T02:52:12.954 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for mgr to start... 2024-06-18T02:52:12.954 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for mgr... 2024-06-18T02:52:13.735 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:13 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/1491035075' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-06-18T02:52:14.016 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-06-18T02:52:14.016 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-06-18T02:52:14.016 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsid": "9e93709a-2d1d-11ef-bc9d-c7b262605968", 2024-06-18T02:52:14.016 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "health": { 2024-06-18T02:52:14.016 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-06-18T02:52:14.016 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-06-18T02:52:14.017 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-06-18T02:52:14.017 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-06-18T02:52:14.017 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-06-18T02:52:14.017 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-06-18T02:52:14.017 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 0 2024-06-18T02:52:14.017 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-06-18T02:52:14.017 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-06-18T02:52:14.017 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "smithi016" 2024-06-18T02:52:14.017 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-06-18T02:52:14.017 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-06-18T02:52:14.017 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "monmap": { 2024-06-18T02:52:14.017 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-18T02:52:14.017 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-06-18T02:52:14.018 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-06-18T02:52:14.018 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-06-18T02:52:14.018 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-06-18T02:52:14.018 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-18T02:52:14.018 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-06-18T02:52:14.018 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-06-18T02:52:14.018 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-06-18T02:52:14.025 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-06-18T02:52:14.025 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-06-18T02:52:14.025 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-06-18T02:52:14.026 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-06-18T02:52:14.026 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-06-18T02:52:14.026 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-06-18T02:52:14.026 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-06-18T02:52:14.026 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-06-18T02:52:14.026 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-06-18T02:52:14.026 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-06-18T02:52:14.026 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-06-18T02:52:14.026 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-06-18T02:52:14.026 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-06-18T02:52:14.026 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-06-18T02:52:14.026 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-06-18T02:52:14.027 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-18T02:52:14.027 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-06-18T02:52:14.027 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-06-18T02:52:14.027 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-06-18T02:52:14.027 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-06-18T02:52:14.027 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "available": false, 2024-06-18T02:52:14.027 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-06-18T02:52:14.027 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modules": [ 2024-06-18T02:52:14.027 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "iostat", 2024-06-18T02:52:14.027 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "nfs", 2024-06-18T02:52:14.027 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "restful" 2024-06-18T02:52:14.027 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-06-18T02:52:14.027 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-18T02:52:14.027 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-06-18T02:52:14.028 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-06-18T02:52:14.028 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-18T02:52:14.028 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modified": "2024-06-18T02:51:57.321261+0000", 2024-06-18T02:52:14.028 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-18T02:52:14.028 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-06-18T02:52:14.028 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-06-18T02:52:14.028 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-06-18T02:52:14.028 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:13.377+0000 7fd59639d640 1 Processor -- start 2024-06-18T02:52:14.028 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:13.377+0000 7fd59639d640 1 -- start start 2024-06-18T02:52:14.028 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:13.377+0000 7fd59639d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd590100af0 0x7fd590100f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:14.028 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:13.377+0000 7fd59639d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd590101440 con 0x7fd590100af0 2024-06-18T02:52:14.029 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:13.378+0000 7fd59539b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd590100af0 0x7fd590100f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:14.029 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:13.378+0000 7fd59539b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd590100af0 0x7fd590100f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45774/0 (socket says 172.21.15.16:45774) 2024-06-18T02:52:14.029 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:13.378+0000 7fd59539b640 1 -- 172.21.15.16:0/1153694436 learned_addr learned my addr 172.21.15.16:0/1153694436 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:52:14.029 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:13.378+0000 7fd59539b640 1 -- 172.21.15.16:0/1153694436 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd590101580 con 0x7fd590100af0 2024-06-18T02:52:14.029 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:13.379+0000 7fd59539b640 1 --2- 172.21.15.16:0/1153694436 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd590100af0 0x7fd590100f00 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fd58c009840 tx=0x7fd58c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a555141053e7eec server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:14.029 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:13.379+0000 7fd587fff640 1 -- 172.21.15.16:0/1153694436 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd58c032440 con 0x7fd590100af0 2024-06-18T02:52:14.029 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:13.379+0000 7fd587fff640 1 -- 172.21.15.16:0/1153694436 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fd58c032a30 con 0x7fd590100af0 2024-06-18T02:52:14.029 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:13.379+0000 7fd587fff640 1 -- 172.21.15.16:0/1153694436 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd58c031ad0 con 0x7fd590100af0 2024-06-18T02:52:14.029 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:13.380+0000 7fd59639d640 1 -- 172.21.15.16:0/1153694436 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd590100af0 msgr2=0x7fd590100f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:14.029 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:13.380+0000 7fd59639d640 1 --2- 172.21.15.16:0/1153694436 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd590100af0 0x7fd590100f00 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fd58c009840 tx=0x7fd58c0091d0 comp rx=0 tx=0).stop 2024-06-18T02:52:14.029 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:13.380+0000 7fd59639d640 1 -- 172.21.15.16:0/1153694436 shutdown_connections 2024-06-18T02:52:14.029 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:13.380+0000 7fd59639d640 1 --2- 172.21.15.16:0/1153694436 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd590100af0 0x7fd590100f00 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:14.030 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:13.380+0000 7fd59639d640 1 -- 172.21.15.16:0/1153694436 >> 172.21.15.16:0/1153694436 conn(0x7fd5900fc270 msgr2=0x7fd5900fe6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:52:14.030 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:13.380+0000 7fd59639d640 1 -- 172.21.15.16:0/1153694436 shutdown_connections 2024-06-18T02:52:14.030 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:13.380+0000 7fd59639d640 1 -- 172.21.15.16:0/1153694436 wait complete. 2024-06-18T02:52:14.030 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:13.380+0000 7fd59639d640 1 Processor -- start 2024-06-18T02:52:14.030 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:13.380+0000 7fd59639d640 1 -- start start 2024-06-18T02:52:14.030 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:13.380+0000 7fd59639d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd590100af0 0x7fd590110960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:14.030 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:13.380+0000 7fd59639d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd590110ea0 con 0x7fd590100af0 2024-06-18T02:52:14.030 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:13.381+0000 7fd59539b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd590100af0 0x7fd590110960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:14.030 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:13.381+0000 7fd59539b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd590100af0 0x7fd590110960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45776/0 (socket says 172.21.15.16:45776) 2024-06-18T02:52:14.030 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:13.381+0000 7fd59539b640 1 -- 172.21.15.16:0/1491035075 learned_addr learned my addr 172.21.15.16:0/1491035075 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:52:14.030 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:13.381+0000 7fd59539b640 1 -- 172.21.15.16:0/1491035075 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd58c009450 con 0x7fd590100af0 2024-06-18T02:52:14.030 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:13.381+0000 7fd59539b640 1 --2- 172.21.15.16:0/1491035075 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd590100af0 0x7fd590110960 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd58c009420 tx=0x7fd58c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:14.030 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:13.381+0000 7fd585ffb640 1 -- 172.21.15.16:0/1491035075 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd58c03d5b0 con 0x7fd590100af0 2024-06-18T02:52:14.031 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:13.381+0000 7fd585ffb640 1 -- 172.21.15.16:0/1491035075 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fd58c03dba0 con 0x7fd590100af0 2024-06-18T02:52:14.031 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:13.381+0000 7fd59639d640 1 -- 172.21.15.16:0/1491035075 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5901110a0 con 0x7fd590100af0 2024-06-18T02:52:14.031 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:13.381+0000 7fd585ffb640 1 -- 172.21.15.16:0/1491035075 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd58c03bb40 con 0x7fd590100af0 2024-06-18T02:52:14.031 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:13.382+0000 7fd59639d640 1 -- 172.21.15.16:0/1491035075 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd590111510 con 0x7fd590100af0 2024-06-18T02:52:14.031 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:13.382+0000 7fd585ffb640 1 -- 172.21.15.16:0/1491035075 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7fd58c039070 con 0x7fd590100af0 2024-06-18T02:52:14.031 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:13.382+0000 7fd585ffb640 1 -- 172.21.15.16:0/1491035075 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fd58c046900 con 0x7fd590100af0 2024-06-18T02:52:14.031 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:13.383+0000 7fd59639d640 1 -- 172.21.15.16:0/1491035075 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd59004b480 con 0x7fd590100af0 2024-06-18T02:52:14.031 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:13.385+0000 7fd585ffb640 1 -- 172.21.15.16:0/1491035075 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71547 (secure 0 0 0) 0x7fd590111510 con 0x7fd590100af0 2024-06-18T02:52:14.031 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:13.449+0000 7fd59639d640 1 -- 172.21.15.16:0/1491035075 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fd59010a510 con 0x7fd590100af0 2024-06-18T02:52:14.031 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:13.449+0000 7fd585ffb640 1 -- 172.21.15.16:0/1491035075 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7fd58c041020 con 0x7fd590100af0 2024-06-18T02:52:14.031 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:13.450+0000 7fd59639d640 1 -- 172.21.15.16:0/1491035075 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd590100af0 msgr2=0x7fd590110960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:14.031 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:13.450+0000 7fd59639d640 1 --2- 172.21.15.16:0/1491035075 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd590100af0 0x7fd590110960 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd58c009420 tx=0x7fd58c008040 comp rx=0 tx=0).stop 2024-06-18T02:52:14.032 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:13.450+0000 7fd59639d640 1 -- 172.21.15.16:0/1491035075 shutdown_connections 2024-06-18T02:52:14.032 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:13.450+0000 7fd59639d640 1 --2- 172.21.15.16:0/1491035075 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd590100af0 0x7fd590110960 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:14.032 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:13.450+0000 7fd59639d640 1 -- 172.21.15.16:0/1491035075 >> 172.21.15.16:0/1491035075 conn(0x7fd5900fc270 msgr2=0x7fd5900fd1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:52:14.032 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:13.450+0000 7fd59639d640 1 -- 172.21.15.16:0/1491035075 shutdown_connections 2024-06-18T02:52:14.032 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:13.450+0000 7fd59639d640 1 -- 172.21.15.16:0/1491035075 wait complete. 2024-06-18T02:52:14.032 INFO:teuthology.orchestra.run.smithi016.stdout:mgr not available, waiting (1/15)... 2024-06-18T02:52:16.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:16 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/3179550423' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-06-18T02:52:17.073 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-06-18T02:52:17.073 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-06-18T02:52:17.073 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsid": "9e93709a-2d1d-11ef-bc9d-c7b262605968", 2024-06-18T02:52:17.073 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "health": { 2024-06-18T02:52:17.073 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-06-18T02:52:17.073 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-06-18T02:52:17.074 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-06-18T02:52:17.074 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-06-18T02:52:17.074 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-06-18T02:52:17.074 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-06-18T02:52:17.074 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 0 2024-06-18T02:52:17.074 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-06-18T02:52:17.074 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-06-18T02:52:17.074 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "smithi016" 2024-06-18T02:52:17.074 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-06-18T02:52:17.074 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-06-18T02:52:17.075 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "monmap": { 2024-06-18T02:52:17.075 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-18T02:52:17.075 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-06-18T02:52:17.075 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-06-18T02:52:17.075 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-06-18T02:52:17.075 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-06-18T02:52:17.075 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-18T02:52:17.075 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-06-18T02:52:17.075 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-06-18T02:52:17.075 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-06-18T02:52:17.075 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-06-18T02:52:17.075 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-06-18T02:52:17.075 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-06-18T02:52:17.076 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-06-18T02:52:17.076 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-06-18T02:52:17.076 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-06-18T02:52:17.076 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-06-18T02:52:17.076 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-06-18T02:52:17.076 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-06-18T02:52:17.076 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-06-18T02:52:17.076 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-06-18T02:52:17.076 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-06-18T02:52:17.076 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-06-18T02:52:17.076 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-06-18T02:52:17.076 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-06-18T02:52:17.077 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-18T02:52:17.077 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-06-18T02:52:17.077 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-06-18T02:52:17.077 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-06-18T02:52:17.077 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-06-18T02:52:17.077 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "available": false, 2024-06-18T02:52:17.077 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-06-18T02:52:17.077 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modules": [ 2024-06-18T02:52:17.077 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "iostat", 2024-06-18T02:52:17.077 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "nfs", 2024-06-18T02:52:17.077 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "restful" 2024-06-18T02:52:17.077 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-06-18T02:52:17.077 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-18T02:52:17.078 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-06-18T02:52:17.078 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-06-18T02:52:17.078 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-18T02:52:17.078 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modified": "2024-06-18T02:51:57.321261+0000", 2024-06-18T02:52:17.078 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-18T02:52:17.078 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-06-18T02:52:17.078 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-06-18T02:52:17.078 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-06-18T02:52:17.078 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:16.423+0000 7fe9ae296640 1 Processor -- start 2024-06-18T02:52:17.078 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:16.423+0000 7fe9ae296640 1 -- start start 2024-06-18T02:52:17.079 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:16.423+0000 7fe9ae296640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9a8075ac0 0x7fe9a8104a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:17.079 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:16.423+0000 7fe9ae296640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a8075ed0 con 0x7fe9a8075ac0 2024-06-18T02:52:17.079 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:16.423+0000 7fe9a7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9a8075ac0 0x7fe9a8104a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:17.079 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:16.423+0000 7fe9a7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9a8075ac0 0x7fe9a8104a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45790/0 (socket says 172.21.15.16:45790) 2024-06-18T02:52:17.079 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:16.423+0000 7fe9a7fff640 1 -- 172.21.15.16:0/3695585691 learned_addr learned my addr 172.21.15.16:0/3695585691 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:52:17.079 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:16.423+0000 7fe9a7fff640 1 -- 172.21.15.16:0/3695585691 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9a8104f60 con 0x7fe9a8075ac0 2024-06-18T02:52:17.079 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:16.424+0000 7fe9a7fff640 1 --2- 172.21.15.16:0/3695585691 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9a8075ac0 0x7fe9a8104a20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe998009840 tx=0x7fe9980091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67da2a78a8cda31a server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:17.079 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:16.425+0000 7fe9a6ffd640 1 -- 172.21.15.16:0/3695585691 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe998032440 con 0x7fe9a8075ac0 2024-06-18T02:52:17.079 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:16.425+0000 7fe9a6ffd640 1 -- 172.21.15.16:0/3695585691 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fe998032a30 con 0x7fe9a8075ac0 2024-06-18T02:52:17.079 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:16.425+0000 7fe9a6ffd640 1 -- 172.21.15.16:0/3695585691 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe998031ad0 con 0x7fe9a8075ac0 2024-06-18T02:52:17.079 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:16.425+0000 7fe9ae296640 1 -- 172.21.15.16:0/3695585691 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9a8075ac0 msgr2=0x7fe9a8104a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:17.079 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:16.425+0000 7fe9ae296640 1 --2- 172.21.15.16:0/3695585691 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9a8075ac0 0x7fe9a8104a20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe998009840 tx=0x7fe9980091d0 comp rx=0 tx=0).stop 2024-06-18T02:52:17.080 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:16.425+0000 7fe9ae296640 1 -- 172.21.15.16:0/3695585691 shutdown_connections 2024-06-18T02:52:17.080 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:16.425+0000 7fe9ae296640 1 --2- 172.21.15.16:0/3695585691 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9a8075ac0 0x7fe9a8104a20 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:17.080 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:16.425+0000 7fe9ae296640 1 -- 172.21.15.16:0/3695585691 >> 172.21.15.16:0/3695585691 conn(0x7fe9a80fc430 msgr2=0x7fe9a80fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:52:17.080 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:16.425+0000 7fe9ae296640 1 -- 172.21.15.16:0/3695585691 shutdown_connections 2024-06-18T02:52:17.080 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:16.425+0000 7fe9ae296640 1 -- 172.21.15.16:0/3695585691 wait complete. 2024-06-18T02:52:17.080 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:16.426+0000 7fe9ae296640 1 Processor -- start 2024-06-18T02:52:17.080 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:16.426+0000 7fe9ae296640 1 -- start start 2024-06-18T02:52:17.080 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:16.426+0000 7fe9ae296640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9a8075ac0 0x7fe9a8102350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:17.080 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:16.426+0000 7fe9ae296640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a8102890 con 0x7fe9a8075ac0 2024-06-18T02:52:17.080 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:16.426+0000 7fe9a7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9a8075ac0 0x7fe9a8102350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:17.080 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:16.426+0000 7fe9a7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9a8075ac0 0x7fe9a8102350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45802/0 (socket says 172.21.15.16:45802) 2024-06-18T02:52:17.080 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:16.426+0000 7fe9a7fff640 1 -- 172.21.15.16:0/3179550423 learned_addr learned my addr 172.21.15.16:0/3179550423 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:52:17.081 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:16.427+0000 7fe9a7fff640 1 -- 172.21.15.16:0/3179550423 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe998009450 con 0x7fe9a8075ac0 2024-06-18T02:52:17.081 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:16.427+0000 7fe9a7fff640 1 --2- 172.21.15.16:0/3179550423 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9a8075ac0 0x7fe9a8102350 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe998031b90 tx=0x7fe998008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:17.081 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:16.427+0000 7fe9a4ff9640 1 -- 172.21.15.16:0/3179550423 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe9980328b0 con 0x7fe9a8075ac0 2024-06-18T02:52:17.081 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:16.427+0000 7fe9a4ff9640 1 -- 172.21.15.16:0/3179550423 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fe99803d8e0 con 0x7fe9a8075ac0 2024-06-18T02:52:17.081 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:16.427+0000 7fe9ae296640 1 -- 172.21.15.16:0/3179550423 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9a8100920 con 0x7fe9a8075ac0 2024-06-18T02:52:17.081 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:16.427+0000 7fe9a4ff9640 1 -- 172.21.15.16:0/3179550423 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe99803ba20 con 0x7fe9a8075ac0 2024-06-18T02:52:17.081 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:16.427+0000 7fe9ae296640 1 -- 172.21.15.16:0/3179550423 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9a8100d90 con 0x7fe9a8075ac0 2024-06-18T02:52:17.081 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:16.427+0000 7fe9a4ff9640 1 -- 172.21.15.16:0/3179550423 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7fe998039070 con 0x7fe9a8075ac0 2024-06-18T02:52:17.081 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:16.427+0000 7fe9a4ff9640 1 -- 172.21.15.16:0/3179550423 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fe998046930 con 0x7fe9a8075ac0 2024-06-18T02:52:17.081 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:16.428+0000 7fe9ae296640 1 -- 172.21.15.16:0/3179550423 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9a804b480 con 0x7fe9a8075ac0 2024-06-18T02:52:17.081 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:16.432+0000 7fe9a4ff9640 1 -- 172.21.15.16:0/3179550423 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71547 (secure 0 0 0) 0x7fe998065790 con 0x7fe9a8075ac0 2024-06-18T02:52:17.081 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:16.496+0000 7fe9ae296640 1 -- 172.21.15.16:0/3179550423 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fe9a8105ce0 con 0x7fe9a8075ac0 2024-06-18T02:52:17.081 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:16.496+0000 7fe9a4ff9640 1 -- 172.21.15.16:0/3179550423 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7fe998054020 con 0x7fe9a8075ac0 2024-06-18T02:52:17.082 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:16.497+0000 7fe9ae296640 1 -- 172.21.15.16:0/3179550423 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9a8075ac0 msgr2=0x7fe9a8102350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:17.082 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:16.497+0000 7fe9ae296640 1 --2- 172.21.15.16:0/3179550423 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9a8075ac0 0x7fe9a8102350 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe998031b90 tx=0x7fe998008040 comp rx=0 tx=0).stop 2024-06-18T02:52:17.082 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:16.497+0000 7fe9ae296640 1 -- 172.21.15.16:0/3179550423 shutdown_connections 2024-06-18T02:52:17.082 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:16.497+0000 7fe9ae296640 1 --2- 172.21.15.16:0/3179550423 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9a8075ac0 0x7fe9a8102350 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:17.082 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:16.497+0000 7fe9ae296640 1 -- 172.21.15.16:0/3179550423 >> 172.21.15.16:0/3179550423 conn(0x7fe9a80fc430 msgr2=0x7fe9a80fcfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:52:17.082 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:16.498+0000 7fe9ae296640 1 -- 172.21.15.16:0/3179550423 shutdown_connections 2024-06-18T02:52:17.082 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:16.498+0000 7fe9ae296640 1 -- 172.21.15.16:0/3179550423 wait complete. 2024-06-18T02:52:17.082 INFO:teuthology.orchestra.run.smithi016.stdout:mgr not available, waiting (2/15)... 2024-06-18T02:52:17.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:17 smithi016 ceph-mon[29214]: Activating manager daemon smithi016.amxydz 2024-06-18T02:52:17.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:17 smithi016 ceph-mon[29214]: mgrmap e2: smithi016.amxydz(active, starting, since 0.0028351s) 2024-06-18T02:52:17.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:17 smithi016 ceph-mon[29214]: from='mgr.14100 172.21.15.16:0/1029075280' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-18T02:52:17.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:17 smithi016 ceph-mon[29214]: from='mgr.14100 172.21.15.16:0/1029075280' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-18T02:52:17.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:17 smithi016 ceph-mon[29214]: from='mgr.14100 172.21.15.16:0/1029075280' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-18T02:52:17.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:17 smithi016 ceph-mon[29214]: from='mgr.14100 172.21.15.16:0/1029075280' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "mon metadata", "id": "smithi016"}]: dispatch 2024-06-18T02:52:17.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:17 smithi016 ceph-mon[29214]: from='mgr.14100 172.21.15.16:0/1029075280' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "mgr metadata", "who": "smithi016.amxydz", "id": "smithi016.amxydz"}]: dispatch 2024-06-18T02:52:17.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:17 smithi016 ceph-mon[29214]: Manager daemon smithi016.amxydz is now available 2024-06-18T02:52:17.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:17 smithi016 ceph-mon[29214]: from='mgr.14100 172.21.15.16:0/1029075280' entity='mgr.smithi016.amxydz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi016.amxydz/mirror_snapshot_schedule"}]: dispatch 2024-06-18T02:52:17.806 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:17 smithi016 ceph-mon[29214]: from='mgr.14100 172.21.15.16:0/1029075280' entity='mgr.smithi016.amxydz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi016.amxydz/trash_purge_schedule"}]: dispatch 2024-06-18T02:52:17.806 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:17 smithi016 ceph-mon[29214]: from='mgr.14100 172.21.15.16:0/1029075280' entity='mgr.smithi016.amxydz' 2024-06-18T02:52:17.806 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:17 smithi016 ceph-mon[29214]: from='mgr.14100 172.21.15.16:0/1029075280' entity='mgr.smithi016.amxydz' 2024-06-18T02:52:17.806 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:17 smithi016 ceph-mon[29214]: from='mgr.14100 172.21.15.16:0/1029075280' entity='mgr.smithi016.amxydz' 2024-06-18T02:52:19.714 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:19 smithi016 ceph-mon[29214]: mgrmap e3: smithi016.amxydz(active, since 1.00705s) 2024-06-18T02:52:20.190 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-06-18T02:52:20.190 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-06-18T02:52:20.190 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsid": "9e93709a-2d1d-11ef-bc9d-c7b262605968", 2024-06-18T02:52:20.191 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "health": { 2024-06-18T02:52:20.191 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-06-18T02:52:20.191 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-06-18T02:52:20.191 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-06-18T02:52:20.191 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-06-18T02:52:20.191 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-06-18T02:52:20.191 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-06-18T02:52:20.191 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 0 2024-06-18T02:52:20.191 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-06-18T02:52:20.191 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-06-18T02:52:20.192 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "smithi016" 2024-06-18T02:52:20.192 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-06-18T02:52:20.192 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-06-18T02:52:20.192 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "monmap": { 2024-06-18T02:52:20.192 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-18T02:52:20.192 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-06-18T02:52:20.192 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-06-18T02:52:20.192 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-06-18T02:52:20.192 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-06-18T02:52:20.192 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-18T02:52:20.192 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-06-18T02:52:20.192 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-06-18T02:52:20.193 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-06-18T02:52:20.193 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-06-18T02:52:20.193 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-06-18T02:52:20.193 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-06-18T02:52:20.193 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-06-18T02:52:20.193 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-06-18T02:52:20.193 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-06-18T02:52:20.193 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-06-18T02:52:20.193 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-06-18T02:52:20.193 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-06-18T02:52:20.193 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-06-18T02:52:20.194 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-06-18T02:52:20.194 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-06-18T02:52:20.194 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-06-18T02:52:20.194 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-06-18T02:52:20.194 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-06-18T02:52:20.194 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-18T02:52:20.194 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-06-18T02:52:20.194 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-06-18T02:52:20.194 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-06-18T02:52:20.194 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-06-18T02:52:20.195 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "available": true, 2024-06-18T02:52:20.195 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-06-18T02:52:20.195 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modules": [ 2024-06-18T02:52:20.195 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "iostat", 2024-06-18T02:52:20.195 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "nfs", 2024-06-18T02:52:20.195 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "restful" 2024-06-18T02:52:20.195 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-06-18T02:52:20.195 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-18T02:52:20.195 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-06-18T02:52:20.195 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-06-18T02:52:20.196 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-18T02:52:20.196 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modified": "2024-06-18T02:51:57.321261+0000", 2024-06-18T02:52:20.196 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-18T02:52:20.196 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-06-18T02:52:20.196 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-06-18T02:52:20.196 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-06-18T02:52:20.196 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:19.458+0000 7f05ffe74640 1 Processor -- start 2024-06-18T02:52:20.196 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:19.458+0000 7f05ffe74640 1 -- start start 2024-06-18T02:52:20.196 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:19.458+0000 7f05ffe74640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05f8075b50 0x7f05f8106dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:20.196 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:19.458+0000 7f05ffe74640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05f8075f60 con 0x7f05f8075b50 2024-06-18T02:52:20.197 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:19.458+0000 7f05fdc10640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05f8075b50 0x7f05f8106dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:20.197 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:19.458+0000 7f05fdc10640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05f8075b50 0x7f05f8106dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45866/0 (socket says 172.21.15.16:45866) 2024-06-18T02:52:20.197 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:19.458+0000 7f05fdc10640 1 -- 172.21.15.16:0/3338102990 learned_addr learned my addr 172.21.15.16:0/3338102990 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:52:20.197 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:19.459+0000 7f05fdc10640 1 -- 172.21.15.16:0/3338102990 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05f81027b0 con 0x7f05f8075b50 2024-06-18T02:52:20.197 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:19.459+0000 7f05fdc10640 1 --2- 172.21.15.16:0/3338102990 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05f8075b50 0x7f05f8106dc0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f05f4009840 tx=0x7f05f40091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f95336f1a735f4 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:20.197 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:19.460+0000 7f05fcc0e640 1 -- 172.21.15.16:0/3338102990 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f05f4032440 con 0x7f05f8075b50 2024-06-18T02:52:20.197 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:19.460+0000 7f05fcc0e640 1 -- 172.21.15.16:0/3338102990 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f05f4032a30 con 0x7f05f8075b50 2024-06-18T02:52:20.197 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:19.460+0000 7f05fcc0e640 1 -- 172.21.15.16:0/3338102990 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f05f4031ad0 con 0x7f05f8075b50 2024-06-18T02:52:20.197 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:19.460+0000 7f05ffe74640 1 -- 172.21.15.16:0/3338102990 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05f8075b50 msgr2=0x7f05f8106dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:20.197 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:19.460+0000 7f05ffe74640 1 --2- 172.21.15.16:0/3338102990 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05f8075b50 0x7f05f8106dc0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f05f4009840 tx=0x7f05f40091d0 comp rx=0 tx=0).stop 2024-06-18T02:52:20.197 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:19.460+0000 7f05ffe74640 1 -- 172.21.15.16:0/3338102990 shutdown_connections 2024-06-18T02:52:20.197 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:19.460+0000 7f05ffe74640 1 --2- 172.21.15.16:0/3338102990 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05f8075b50 0x7f05f8106dc0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:20.197 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:19.460+0000 7f05ffe74640 1 -- 172.21.15.16:0/3338102990 >> 172.21.15.16:0/3338102990 conn(0x7f05f80fc4b0 msgr2=0x7f05f80fe8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:52:20.197 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:19.460+0000 7f05ffe74640 1 -- 172.21.15.16:0/3338102990 shutdown_connections 2024-06-18T02:52:20.198 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:19.460+0000 7f05ffe74640 1 -- 172.21.15.16:0/3338102990 wait complete. 2024-06-18T02:52:20.198 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:19.461+0000 7f05ffe74640 1 Processor -- start 2024-06-18T02:52:20.198 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:19.461+0000 7f05ffe74640 1 -- start start 2024-06-18T02:52:20.198 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:19.461+0000 7f05ffe74640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05f8075b50 0x7f05f8112c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:20.198 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:19.461+0000 7f05ffe74640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05f81131a0 con 0x7f05f8075b50 2024-06-18T02:52:20.198 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:19.461+0000 7f05fdc10640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05f8075b50 0x7f05f8112c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:20.198 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:19.461+0000 7f05fdc10640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05f8075b50 0x7f05f8112c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45868/0 (socket says 172.21.15.16:45868) 2024-06-18T02:52:20.198 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:19.461+0000 7f05fdc10640 1 -- 172.21.15.16:0/455030128 learned_addr learned my addr 172.21.15.16:0/455030128 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:52:20.198 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:19.462+0000 7f05fdc10640 1 -- 172.21.15.16:0/455030128 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05f4009450 con 0x7f05f8075b50 2024-06-18T02:52:20.198 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:19.462+0000 7f05fdc10640 1 --2- 172.21.15.16:0/455030128 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05f8075b50 0x7f05f8112c60 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f05f4006f00 tx=0x7f05f4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:20.198 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:19.462+0000 7f05ee7fc640 1 -- 172.21.15.16:0/455030128 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f05f403d450 con 0x7f05f8075b50 2024-06-18T02:52:20.198 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:19.462+0000 7f05ee7fc640 1 -- 172.21.15.16:0/455030128 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f05f403da40 con 0x7f05f8075b50 2024-06-18T02:52:20.198 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:19.462+0000 7f05ffe74640 1 -- 172.21.15.16:0/455030128 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05f8113400 con 0x7f05f8075b50 2024-06-18T02:52:20.199 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:19.462+0000 7f05ee7fc640 1 -- 172.21.15.16:0/455030128 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f05f403bb10 con 0x7f05f8075b50 2024-06-18T02:52:20.199 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:19.462+0000 7f05ffe74640 1 -- 172.21.15.16:0/455030128 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05f8113f30 con 0x7f05f8075b50 2024-06-18T02:52:20.199 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:19.463+0000 7f05ee7fc640 1 -- 172.21.15.16:0/455030128 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 41829+0+0 (secure 0 0 0) 0x7f05f4039070 con 0x7f05f8075b50 2024-06-18T02:52:20.199 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:19.463+0000 7f05ee7fc640 1 --2- 172.21.15.16:0/455030128 >> [v2:172.21.15.16:6800/1678074233,v1:172.21.15.16:6801/1678074233] conn(0x7f05e40346e0 0x7f05e4036b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:20.199 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:19.463+0000 7f05ee7fc640 1 -- 172.21.15.16:0/455030128 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f05f4067160 con 0x7f05f8075b50 2024-06-18T02:52:20.199 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:19.463+0000 7f05ffe74640 1 -- 172.21.15.16:0/455030128 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05f805ead0 con 0x7f05f8075b50 2024-06-18T02:52:20.199 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:19.463+0000 7f05fd40f640 1 --2- 172.21.15.16:0/455030128 >> [v2:172.21.15.16:6800/1678074233,v1:172.21.15.16:6801/1678074233] conn(0x7f05e40346e0 0x7f05e4036b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:20.199 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:19.464+0000 7f05fd40f640 1 --2- 172.21.15.16:0/455030128 >> [v2:172.21.15.16:6800/1678074233,v1:172.21.15.16:6801/1678074233] conn(0x7f05e40346e0 0x7f05e4036b90 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f05e8006fd0 tx=0x7f05e8006db0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:20.200 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:19.469+0000 7f05ee7fc640 1 -- 172.21.15.16:0/455030128 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f05f4041020 con 0x7f05f8075b50 2024-06-18T02:52:20.200 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:19.627+0000 7f05ee7fc640 1 -- 172.21.15.16:0/455030128 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 4) v1 ==== 41935+0+0 (secure 0 0 0) 0x7f05f4046c50 con 0x7f05f8075b50 2024-06-18T02:52:20.200 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:19.712+0000 7f05ffe74640 1 -- 172.21.15.16:0/455030128 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f05f8107f20 con 0x7f05f8075b50 2024-06-18T02:52:20.200 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:19.713+0000 7f05ee7fc640 1 -- 172.21.15.16:0/455030128 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1247 (secure 0 0 0) 0x7f05f4037020 con 0x7f05f8075b50 2024-06-18T02:52:20.200 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:19.715+0000 7f05ffe74640 1 -- 172.21.15.16:0/455030128 >> [v2:172.21.15.16:6800/1678074233,v1:172.21.15.16:6801/1678074233] conn(0x7f05e40346e0 msgr2=0x7f05e4036b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:20.200 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:19.715+0000 7f05ffe74640 1 --2- 172.21.15.16:0/455030128 >> [v2:172.21.15.16:6800/1678074233,v1:172.21.15.16:6801/1678074233] conn(0x7f05e40346e0 0x7f05e4036b90 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f05e8006fd0 tx=0x7f05e8006db0 comp rx=0 tx=0).stop 2024-06-18T02:52:20.200 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:19.715+0000 7f05ffe74640 1 -- 172.21.15.16:0/455030128 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05f8075b50 msgr2=0x7f05f8112c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:20.200 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:19.715+0000 7f05ffe74640 1 --2- 172.21.15.16:0/455030128 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05f8075b50 0x7f05f8112c60 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f05f4006f00 tx=0x7f05f4008040 comp rx=0 tx=0).stop 2024-06-18T02:52:20.200 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:19.715+0000 7f05ffe74640 1 -- 172.21.15.16:0/455030128 shutdown_connections 2024-06-18T02:52:20.200 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:19.715+0000 7f05ffe74640 1 --2- 172.21.15.16:0/455030128 >> [v2:172.21.15.16:6800/1678074233,v1:172.21.15.16:6801/1678074233] conn(0x7f05e40346e0 0x7f05e4036b90 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:20.200 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:19.715+0000 7f05ffe74640 1 --2- 172.21.15.16:0/455030128 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05f8075b50 0x7f05f8112c60 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:20.200 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:19.715+0000 7f05ffe74640 1 -- 172.21.15.16:0/455030128 >> 172.21.15.16:0/455030128 conn(0x7f05f80fc4b0 msgr2=0x7f05f80fcfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:52:20.200 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:19.715+0000 7f05ffe74640 1 -- 172.21.15.16:0/455030128 shutdown_connections 2024-06-18T02:52:20.201 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:19.715+0000 7f05ffe74640 1 -- 172.21.15.16:0/455030128 wait complete. 2024-06-18T02:52:20.201 INFO:teuthology.orchestra.run.smithi016.stdout:mgr is available 2024-06-18T02:52:20.757 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:20 smithi016 ceph-mon[29214]: mgrmap e4: smithi016.amxydz(active, since 2s) 2024-06-18T02:52:20.757 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:20 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/455030128' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-06-18T02:52:21.308 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-06-18T02:52:21.309 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout [global] 2024-06-18T02:52:21.309 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout fsid = 9e93709a-2d1d-11ef-bc9d-c7b262605968 2024-06-18T02:52:21.309 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-06-18T02:52:21.309 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-06-18T02:52:21.309 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-06-18T02:52:21.309 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-06-18T02:52:21.309 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-06-18T02:52:21.309 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout [mgr] 2024-06-18T02:52:21.310 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-06-18T02:52:21.310 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-06-18T02:52:21.310 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout [osd] 2024-06-18T02:52:21.310 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-06-18T02:52:21.310 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-06-18T02:52:21.310 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-06-18T02:52:21.310 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-06-18T02:52:21.310 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-06-18T02:52:21.310 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:20.572+0000 7f5016558640 1 Processor -- start 2024-06-18T02:52:21.310 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:20.572+0000 7f5016558640 1 -- start start 2024-06-18T02:52:21.310 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:20.572+0000 7f5016558640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5010077c20 0x7f5010076240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:21.310 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:20.572+0000 7f5016558640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5010078030 con 0x7f5010077c20 2024-06-18T02:52:21.310 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:20.573+0000 7f500ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5010077c20 0x7f5010076240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:21.311 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:20.573+0000 7f500ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5010077c20 0x7f5010076240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45878/0 (socket says 172.21.15.16:45878) 2024-06-18T02:52:21.311 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:20.573+0000 7f500ffff640 1 -- 172.21.15.16:0/2730873936 learned_addr learned my addr 172.21.15.16:0/2730873936 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:52:21.311 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:20.573+0000 7f500ffff640 1 -- 172.21.15.16:0/2730873936 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5010076780 con 0x7f5010077c20 2024-06-18T02:52:21.311 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:20.574+0000 7f500ffff640 1 --2- 172.21.15.16:0/2730873936 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5010077c20 0x7f5010076240 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5000009840 tx=0x7f50000091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6e2457a26f88db2 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:21.311 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:20.574+0000 7f500effd640 1 -- 172.21.15.16:0/2730873936 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5000032440 con 0x7f5010077c20 2024-06-18T02:52:21.311 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:20.574+0000 7f500effd640 1 -- 172.21.15.16:0/2730873936 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f5000032a30 con 0x7f5010077c20 2024-06-18T02:52:21.311 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:20.574+0000 7f500effd640 1 -- 172.21.15.16:0/2730873936 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5000031ad0 con 0x7f5010077c20 2024-06-18T02:52:21.311 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:20.574+0000 7f5016558640 1 -- 172.21.15.16:0/2730873936 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5010077c20 msgr2=0x7f5010076240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:21.311 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:20.574+0000 7f5016558640 1 --2- 172.21.15.16:0/2730873936 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5010077c20 0x7f5010076240 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5000009840 tx=0x7f50000091d0 comp rx=0 tx=0).stop 2024-06-18T02:52:21.311 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:20.574+0000 7f5016558640 1 -- 172.21.15.16:0/2730873936 shutdown_connections 2024-06-18T02:52:21.311 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:20.574+0000 7f5016558640 1 --2- 172.21.15.16:0/2730873936 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5010077c20 0x7f5010076240 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:21.311 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:20.574+0000 7f5016558640 1 -- 172.21.15.16:0/2730873936 >> 172.21.15.16:0/2730873936 conn(0x7f50100fd5b0 msgr2=0x7f50100ff9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:52:21.312 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:20.574+0000 7f5016558640 1 -- 172.21.15.16:0/2730873936 shutdown_connections 2024-06-18T02:52:21.312 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:20.574+0000 7f5016558640 1 -- 172.21.15.16:0/2730873936 wait complete. 2024-06-18T02:52:21.312 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:20.575+0000 7f5016558640 1 Processor -- start 2024-06-18T02:52:21.312 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:20.575+0000 7f5016558640 1 -- start start 2024-06-18T02:52:21.312 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:20.575+0000 7f5016558640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5010077c20 0x7f5010113030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:21.312 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:20.575+0000 7f5016558640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5010113570 con 0x7f5010077c20 2024-06-18T02:52:21.312 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:20.575+0000 7f500ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5010077c20 0x7f5010113030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:21.312 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:20.575+0000 7f500ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5010077c20 0x7f5010113030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45888/0 (socket says 172.21.15.16:45888) 2024-06-18T02:52:21.312 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:20.575+0000 7f500ffff640 1 -- 172.21.15.16:0/2139025346 learned_addr learned my addr 172.21.15.16:0/2139025346 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:52:21.312 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:20.576+0000 7f500ffff640 1 -- 172.21.15.16:0/2139025346 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5000009450 con 0x7f5010077c20 2024-06-18T02:52:21.313 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:20.576+0000 7f500ffff640 1 --2- 172.21.15.16:0/2139025346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5010077c20 0x7f5010113030 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5000031b90 tx=0x7f5000008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:21.313 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:20.576+0000 7f500cff9640 1 -- 172.21.15.16:0/2139025346 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f50000328b0 con 0x7f5010077c20 2024-06-18T02:52:21.313 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:20.576+0000 7f500cff9640 1 -- 172.21.15.16:0/2139025346 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f500003d8e0 con 0x7f5010077c20 2024-06-18T02:52:21.313 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:20.576+0000 7f5016558640 1 -- 172.21.15.16:0/2139025346 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5010113770 con 0x7f5010077c20 2024-06-18T02:52:21.313 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:20.576+0000 7f500cff9640 1 -- 172.21.15.16:0/2139025346 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f500003ba20 con 0x7f5010077c20 2024-06-18T02:52:21.313 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:20.576+0000 7f5016558640 1 -- 172.21.15.16:0/2139025346 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5010113be0 con 0x7f5010077c20 2024-06-18T02:52:21.313 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:20.577+0000 7f500cff9640 1 -- 172.21.15.16:0/2139025346 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 4) v1 ==== 41935+0+0 (secure 0 0 0) 0x7f5000039070 con 0x7f5010077c20 2024-06-18T02:52:21.313 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:20.577+0000 7f500cff9640 1 --2- 172.21.15.16:0/2139025346 >> [v2:172.21.15.16:6800/1678074233,v1:172.21.15.16:6801/1678074233] conn(0x7f4ff8034810 0x7f4ff8036cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:21.313 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:20.577+0000 7f500cff9640 1 -- 172.21.15.16:0/2139025346 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f5000067360 con 0x7f5010077c20 2024-06-18T02:52:21.313 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:20.577+0000 7f5016558640 1 -- 172.21.15.16:0/2139025346 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f501005ead0 con 0x7f5010077c20 2024-06-18T02:52:21.313 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:20.577+0000 7f500f7fe640 1 --2- 172.21.15.16:0/2139025346 >> [v2:172.21.15.16:6800/1678074233,v1:172.21.15.16:6801/1678074233] conn(0x7f4ff8034810 0x7f4ff8036cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:21.314 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:20.578+0000 7f500f7fe640 1 --2- 172.21.15.16:0/2139025346 >> [v2:172.21.15.16:6800/1678074233,v1:172.21.15.16:6801/1678074233] conn(0x7f4ff8034810 0x7f4ff8036cc0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5004006fd0 tx=0x7f5004006db0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:21.314 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:20.582+0000 7f500cff9640 1 -- 172.21.15.16:0/2139025346 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f5000041070 con 0x7f5010077c20 2024-06-18T02:52:21.314 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:20.755+0000 7f5016558640 1 -- 172.21.15.16:0/2139025346 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f501004b480 con 0x7f5010077c20 2024-06-18T02:52:21.314 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:20.756+0000 7f500cff9640 1 -- 172.21.15.16:0/2139025346 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7f5000037030 con 0x7f5010077c20 2024-06-18T02:52:21.314 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:20.758+0000 7f5016558640 1 -- 172.21.15.16:0/2139025346 >> [v2:172.21.15.16:6800/1678074233,v1:172.21.15.16:6801/1678074233] conn(0x7f4ff8034810 msgr2=0x7f4ff8036cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:21.314 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:20.758+0000 7f5016558640 1 --2- 172.21.15.16:0/2139025346 >> [v2:172.21.15.16:6800/1678074233,v1:172.21.15.16:6801/1678074233] conn(0x7f4ff8034810 0x7f4ff8036cc0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5004006fd0 tx=0x7f5004006db0 comp rx=0 tx=0).stop 2024-06-18T02:52:21.314 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:20.758+0000 7f5016558640 1 -- 172.21.15.16:0/2139025346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5010077c20 msgr2=0x7f5010113030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:21.314 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:20.758+0000 7f5016558640 1 --2- 172.21.15.16:0/2139025346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5010077c20 0x7f5010113030 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5000031b90 tx=0x7f5000008040 comp rx=0 tx=0).stop 2024-06-18T02:52:21.315 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:20.758+0000 7f5016558640 1 -- 172.21.15.16:0/2139025346 shutdown_connections 2024-06-18T02:52:21.315 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:20.758+0000 7f5016558640 1 --2- 172.21.15.16:0/2139025346 >> [v2:172.21.15.16:6800/1678074233,v1:172.21.15.16:6801/1678074233] conn(0x7f4ff8034810 0x7f4ff8036cc0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:21.315 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:20.758+0000 7f5016558640 1 --2- 172.21.15.16:0/2139025346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5010077c20 0x7f5010113030 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:21.315 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:20.758+0000 7f5016558640 1 -- 172.21.15.16:0/2139025346 >> 172.21.15.16:0/2139025346 conn(0x7f50100fd5b0 msgr2=0x7f50100fe330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:52:21.315 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:20.758+0000 7f5016558640 1 -- 172.21.15.16:0/2139025346 shutdown_connections 2024-06-18T02:52:21.315 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:20.758+0000 7f5016558640 1 -- 172.21.15.16:0/2139025346 wait complete. 2024-06-18T02:52:21.315 INFO:teuthology.orchestra.run.smithi016.stdout:Enabling cephadm module... 2024-06-18T02:52:21.743 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:21 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/2139025346' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-06-18T02:52:22.909 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:22 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/3613717112' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-06-18T02:52:23.172 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:21.745+0000 7f43b5625640 1 Processor -- start 2024-06-18T02:52:23.173 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:21.745+0000 7f43b5625640 1 -- start start 2024-06-18T02:52:23.173 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:21.745+0000 7f43b5625640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43b0077c20 0x7f43b0076240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:23.173 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:21.745+0000 7f43b5625640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b0078030 con 0x7f43b0077c20 2024-06-18T02:52:23.173 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:21.745+0000 7f43aeffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43b0077c20 0x7f43b0076240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:23.173 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:21.745+0000 7f43aeffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43b0077c20 0x7f43b0076240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39222/0 (socket says 172.21.15.16:39222) 2024-06-18T02:52:23.173 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:21.745+0000 7f43aeffd640 1 -- 172.21.15.16:0/1532392777 learned_addr learned my addr 172.21.15.16:0/1532392777 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:52:23.173 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:21.745+0000 7f43aeffd640 1 -- 172.21.15.16:0/1532392777 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43b0076780 con 0x7f43b0077c20 2024-06-18T02:52:23.173 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:21.746+0000 7f43aeffd640 1 --2- 172.21.15.16:0/1532392777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43b0077c20 0x7f43b0076240 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f43a0009840 tx=0x7f43a00091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ebfc8c6a72e5021 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:23.173 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:21.746+0000 7f43ae7fc640 1 -- 172.21.15.16:0/1532392777 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f43a0032440 con 0x7f43b0077c20 2024-06-18T02:52:23.173 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:21.746+0000 7f43ae7fc640 1 -- 172.21.15.16:0/1532392777 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f43a0032a30 con 0x7f43b0077c20 2024-06-18T02:52:23.173 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:21.746+0000 7f43ae7fc640 1 -- 172.21.15.16:0/1532392777 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f43a0031ad0 con 0x7f43b0077c20 2024-06-18T02:52:23.173 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:21.747+0000 7f43b5625640 1 -- 172.21.15.16:0/1532392777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43b0077c20 msgr2=0x7f43b0076240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:23.174 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:21.747+0000 7f43b5625640 1 --2- 172.21.15.16:0/1532392777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43b0077c20 0x7f43b0076240 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f43a0009840 tx=0x7f43a00091d0 comp rx=0 tx=0).stop 2024-06-18T02:52:23.174 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:21.747+0000 7f43b5625640 1 -- 172.21.15.16:0/1532392777 shutdown_connections 2024-06-18T02:52:23.174 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:21.747+0000 7f43b5625640 1 --2- 172.21.15.16:0/1532392777 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43b0077c20 0x7f43b0076240 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:23.174 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:21.747+0000 7f43b5625640 1 -- 172.21.15.16:0/1532392777 >> 172.21.15.16:0/1532392777 conn(0x7f43b00fd5b0 msgr2=0x7f43b00ff9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:52:23.174 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:21.747+0000 7f43b5625640 1 -- 172.21.15.16:0/1532392777 shutdown_connections 2024-06-18T02:52:23.174 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:21.747+0000 7f43b5625640 1 -- 172.21.15.16:0/1532392777 wait complete. 2024-06-18T02:52:23.174 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:21.748+0000 7f43b5625640 1 Processor -- start 2024-06-18T02:52:23.175 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:21.748+0000 7f43b5625640 1 -- start start 2024-06-18T02:52:23.175 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:21.748+0000 7f43b5625640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43b0077c20 0x7f43b0113030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:23.175 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:21.748+0000 7f43b5625640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b0113570 con 0x7f43b0077c20 2024-06-18T02:52:23.175 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:21.748+0000 7f43aeffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43b0077c20 0x7f43b0113030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:23.175 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:21.748+0000 7f43aeffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43b0077c20 0x7f43b0113030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39230/0 (socket says 172.21.15.16:39230) 2024-06-18T02:52:23.175 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:21.748+0000 7f43aeffd640 1 -- 172.21.15.16:0/3613717112 learned_addr learned my addr 172.21.15.16:0/3613717112 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:52:23.175 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:21.748+0000 7f43aeffd640 1 -- 172.21.15.16:0/3613717112 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43a0009450 con 0x7f43b0077c20 2024-06-18T02:52:23.175 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:21.749+0000 7f43aeffd640 1 --2- 172.21.15.16:0/3613717112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43b0077c20 0x7f43b0113030 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f43a0031b90 tx=0x7f43a0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:23.175 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:21.749+0000 7f43a77fe640 1 -- 172.21.15.16:0/3613717112 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f43a00328b0 con 0x7f43b0077c20 2024-06-18T02:52:23.175 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:21.749+0000 7f43a77fe640 1 -- 172.21.15.16:0/3613717112 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f43a003d8e0 con 0x7f43b0077c20 2024-06-18T02:52:23.175 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:21.749+0000 7f43b5625640 1 -- 172.21.15.16:0/3613717112 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43b0113770 con 0x7f43b0077c20 2024-06-18T02:52:23.175 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:21.749+0000 7f43a77fe640 1 -- 172.21.15.16:0/3613717112 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f43a003ba20 con 0x7f43b0077c20 2024-06-18T02:52:23.175 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:21.749+0000 7f43b5625640 1 -- 172.21.15.16:0/3613717112 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43b0113be0 con 0x7f43b0077c20 2024-06-18T02:52:23.176 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:21.750+0000 7f43a77fe640 1 -- 172.21.15.16:0/3613717112 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 4) v1 ==== 41935+0+0 (secure 0 0 0) 0x7f43a0039070 con 0x7f43b0077c20 2024-06-18T02:52:23.176 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:21.750+0000 7f43a77fe640 1 --2- 172.21.15.16:0/3613717112 >> [v2:172.21.15.16:6800/1678074233,v1:172.21.15.16:6801/1678074233] conn(0x7f439c034810 0x7f439c036cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:23.176 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:21.750+0000 7f43a77fe640 1 -- 172.21.15.16:0/3613717112 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f43a00672f0 con 0x7f43b0077c20 2024-06-18T02:52:23.176 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:21.750+0000 7f43b5625640 1 -- 172.21.15.16:0/3613717112 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43b005ead0 con 0x7f43b0077c20 2024-06-18T02:52:23.176 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:21.750+0000 7f43a7fff640 1 --2- 172.21.15.16:0/3613717112 >> [v2:172.21.15.16:6800/1678074233,v1:172.21.15.16:6801/1678074233] conn(0x7f439c034810 0x7f439c036cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:23.176 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:21.751+0000 7f43a7fff640 1 --2- 172.21.15.16:0/3613717112 >> [v2:172.21.15.16:6800/1678074233,v1:172.21.15.16:6801/1678074233] conn(0x7f439c034810 0x7f439c036cc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4398006fd0 tx=0x7f4398006db0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:23.176 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:21.755+0000 7f43a77fe640 1 -- 172.21.15.16:0/3613717112 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f43a0041020 con 0x7f43b0077c20 2024-06-18T02:52:23.177 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:21.975+0000 7f43b5625640 1 -- 172.21.15.16:0/3613717112 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f43b0107f50 con 0x7f43b0077c20 2024-06-18T02:52:23.177 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:22.631+0000 7f43a77fe640 1 -- 172.21.15.16:0/3613717112 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 5) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f43a0045c90 con 0x7f43b0077c20 2024-06-18T02:52:23.177 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:22.633+0000 7f43a77fe640 1 -- 172.21.15.16:0/3613717112 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f43a0066220 con 0x7f43b0077c20 2024-06-18T02:52:23.177 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:22.635+0000 7f43b5625640 1 -- 172.21.15.16:0/3613717112 >> [v2:172.21.15.16:6800/1678074233,v1:172.21.15.16:6801/1678074233] conn(0x7f439c034810 msgr2=0x7f439c036cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:23.177 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:22.635+0000 7f43b5625640 1 --2- 172.21.15.16:0/3613717112 >> [v2:172.21.15.16:6800/1678074233,v1:172.21.15.16:6801/1678074233] conn(0x7f439c034810 0x7f439c036cc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4398006fd0 tx=0x7f4398006db0 comp rx=0 tx=0).stop 2024-06-18T02:52:23.177 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:22.635+0000 7f43b5625640 1 -- 172.21.15.16:0/3613717112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43b0077c20 msgr2=0x7f43b0113030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:23.177 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:22.635+0000 7f43b5625640 1 --2- 172.21.15.16:0/3613717112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43b0077c20 0x7f43b0113030 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f43a0031b90 tx=0x7f43a0008040 comp rx=0 tx=0).stop 2024-06-18T02:52:23.177 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:22.635+0000 7f43b5625640 1 -- 172.21.15.16:0/3613717112 shutdown_connections 2024-06-18T02:52:23.177 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:22.635+0000 7f43b5625640 1 --2- 172.21.15.16:0/3613717112 >> [v2:172.21.15.16:6800/1678074233,v1:172.21.15.16:6801/1678074233] conn(0x7f439c034810 0x7f439c036cc0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:23.177 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:22.635+0000 7f43b5625640 1 --2- 172.21.15.16:0/3613717112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43b0077c20 0x7f43b0113030 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:23.177 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:22.635+0000 7f43b5625640 1 -- 172.21.15.16:0/3613717112 >> 172.21.15.16:0/3613717112 conn(0x7f43b00fd5b0 msgr2=0x7f43b00fe330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:52:23.177 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:22.635+0000 7f43b5625640 1 -- 172.21.15.16:0/3613717112 shutdown_connections 2024-06-18T02:52:23.178 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:22.635+0000 7f43b5625640 1 -- 172.21.15.16:0/3613717112 wait complete. 2024-06-18T02:52:23.766 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:23 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/3613717112' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-06-18T02:52:23.766 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:23 smithi016 ceph-mon[29214]: mgrmap e5: smithi016.amxydz(active, since 5s) 2024-06-18T02:52:24.388 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-06-18T02:52:24.388 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-06-18T02:52:24.388 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "available": true, 2024-06-18T02:52:24.388 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "active_name": "smithi016.amxydz", 2024-06-18T02:52:24.389 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-06-18T02:52:24.389 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-06-18T02:52:24.389 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:23.559+0000 7f9e6705c640 1 Processor -- start 2024-06-18T02:52:24.389 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:23.559+0000 7f9e6705c640 1 -- start start 2024-06-18T02:52:24.389 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:23.559+0000 7f9e6705c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e60075ba0 0x7f9e60104940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:24.389 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:23.559+0000 7f9e6705c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e60104e80 con 0x7f9e60075ba0 2024-06-18T02:52:24.389 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:23.559+0000 7f9e64df8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e60075ba0 0x7f9e60104940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:24.389 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:23.559+0000 7f9e64df8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e60075ba0 0x7f9e60104940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39264/0 (socket says 172.21.15.16:39264) 2024-06-18T02:52:24.389 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:23.560+0000 7f9e64df8640 1 -- 172.21.15.16:0/233666814 learned_addr learned my addr 172.21.15.16:0/233666814 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:52:24.389 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:23.560+0000 7f9e64df8640 1 -- 172.21.15.16:0/233666814 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e60104fc0 con 0x7f9e60075ba0 2024-06-18T02:52:24.389 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:23.560+0000 7f9e64df8640 1 --2- 172.21.15.16:0/233666814 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e60075ba0 0x7f9e60104940 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f9e54009840 tx=0x7f9e540091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c04d7756ba79183f server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:24.390 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:23.561+0000 7f9e5f7fe640 1 -- 172.21.15.16:0/233666814 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e54032440 con 0x7f9e60075ba0 2024-06-18T02:52:24.390 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:23.561+0000 7f9e5f7fe640 1 -- 172.21.15.16:0/233666814 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f9e54032a30 con 0x7f9e60075ba0 2024-06-18T02:52:24.390 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:23.561+0000 7f9e5f7fe640 1 -- 172.21.15.16:0/233666814 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e54031ad0 con 0x7f9e60075ba0 2024-06-18T02:52:24.390 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:23.562+0000 7f9e6705c640 1 -- 172.21.15.16:0/233666814 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e60075ba0 msgr2=0x7f9e60104940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:24.390 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:23.562+0000 7f9e6705c640 1 --2- 172.21.15.16:0/233666814 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e60075ba0 0x7f9e60104940 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f9e54009840 tx=0x7f9e540091d0 comp rx=0 tx=0).stop 2024-06-18T02:52:24.390 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:23.562+0000 7f9e6705c640 1 -- 172.21.15.16:0/233666814 shutdown_connections 2024-06-18T02:52:24.390 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:23.562+0000 7f9e6705c640 1 --2- 172.21.15.16:0/233666814 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e60075ba0 0x7f9e60104940 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:24.390 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:23.562+0000 7f9e6705c640 1 -- 172.21.15.16:0/233666814 >> 172.21.15.16:0/233666814 conn(0x7f9e600fc430 msgr2=0x7f9e600fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:52:24.390 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:23.562+0000 7f9e6705c640 1 -- 172.21.15.16:0/233666814 shutdown_connections 2024-06-18T02:52:24.390 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:23.562+0000 7f9e6705c640 1 -- 172.21.15.16:0/233666814 wait complete. 2024-06-18T02:52:24.390 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:23.562+0000 7f9e6705c640 1 Processor -- start 2024-06-18T02:52:24.390 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:23.562+0000 7f9e6705c640 1 -- start start 2024-06-18T02:52:24.390 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:23.563+0000 7f9e6705c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e60075ba0 0x7f9e60110b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:24.391 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:23.563+0000 7f9e6705c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e60111050 con 0x7f9e60075ba0 2024-06-18T02:52:24.391 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:23.563+0000 7f9e64df8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e60075ba0 0x7f9e60110b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:24.391 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:23.563+0000 7f9e64df8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e60075ba0 0x7f9e60110b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39274/0 (socket says 172.21.15.16:39274) 2024-06-18T02:52:24.391 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:23.563+0000 7f9e64df8640 1 -- 172.21.15.16:0/3651328630 learned_addr learned my addr 172.21.15.16:0/3651328630 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:52:24.391 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:23.563+0000 7f9e64df8640 1 -- 172.21.15.16:0/3651328630 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e54009450 con 0x7f9e60075ba0 2024-06-18T02:52:24.391 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:23.563+0000 7f9e64df8640 1 --2- 172.21.15.16:0/3651328630 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e60075ba0 0x7f9e60110b10 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9e54031b90 tx=0x7f9e54008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:24.391 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:23.564+0000 7f9e5d7fa640 1 -- 172.21.15.16:0/3651328630 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e540328b0 con 0x7f9e60075ba0 2024-06-18T02:52:24.391 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:23.564+0000 7f9e5d7fa640 1 -- 172.21.15.16:0/3651328630 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f9e5403d8e0 con 0x7f9e60075ba0 2024-06-18T02:52:24.391 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:23.564+0000 7f9e6705c640 1 -- 172.21.15.16:0/3651328630 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e60111250 con 0x7f9e60075ba0 2024-06-18T02:52:24.391 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:23.564+0000 7f9e5d7fa640 1 -- 172.21.15.16:0/3651328630 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e5403ba20 con 0x7f9e60075ba0 2024-06-18T02:52:24.391 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:23.564+0000 7f9e6705c640 1 -- 172.21.15.16:0/3651328630 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e601116c0 con 0x7f9e60075ba0 2024-06-18T02:52:24.391 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:23.564+0000 7f9e5d7fa640 1 -- 172.21.15.16:0/3651328630 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f9e54039070 con 0x7f9e60075ba0 2024-06-18T02:52:24.391 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:23.565+0000 7f9e5d7fa640 1 --2- 172.21.15.16:0/3651328630 >> [v2:172.21.15.16:6800/1678074233,v1:172.21.15.16:6801/1678074233] conn(0x7f9e50034860 0x7f9e50036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:24.391 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:23.565+0000 7f9e5d7fa640 1 -- 172.21.15.16:0/3651328630 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f9e54068250 con 0x7f9e60075ba0 2024-06-18T02:52:24.392 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:23.565+0000 7f9e5ffff640 1 -- 172.21.15.16:0/3651328630 >> [v2:172.21.15.16:6800/1678074233,v1:172.21.15.16:6801/1678074233] conn(0x7f9e50034860 msgr2=0x7f9e50036d10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/1678074233 2024-06-18T02:52:24.392 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:23.565+0000 7f9e6705c640 1 -- 172.21.15.16:0/3651328630 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e601116c0 con 0x7f9e60075ba0 2024-06-18T02:52:24.392 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:23.565+0000 7f9e5ffff640 1 --2- 172.21.15.16:0/3651328630 >> [v2:172.21.15.16:6800/1678074233,v1:172.21.15.16:6801/1678074233] conn(0x7f9e50034860 0x7f9e50036d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-18T02:52:24.392 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:23.569+0000 7f9e5d7fa640 1 -- 172.21.15.16:0/3651328630 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f9e601116c0 con 0x7f9e60075ba0 2024-06-18T02:52:24.392 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:23.766+0000 7f9e5ffff640 1 -- 172.21.15.16:0/3651328630 >> [v2:172.21.15.16:6800/1678074233,v1:172.21.15.16:6801/1678074233] conn(0x7f9e50034860 msgr2=0x7f9e50036d10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/1678074233 2024-06-18T02:52:24.392 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:23.766+0000 7f9e5ffff640 1 --2- 172.21.15.16:0/3651328630 >> [v2:172.21.15.16:6800/1678074233,v1:172.21.15.16:6801/1678074233] conn(0x7f9e50034860 0x7f9e50036d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-18T02:52:24.392 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:23.789+0000 7f9e6705c640 1 -- 172.21.15.16:0/3651328630 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f9e601116c0 con 0x7f9e60075ba0 2024-06-18T02:52:24.392 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:23.789+0000 7f9e5d7fa640 1 -- 172.21.15.16:0/3651328630 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7f9e54041020 con 0x7f9e60075ba0 2024-06-18T02:52:24.392 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:23.791+0000 7f9e6705c640 1 -- 172.21.15.16:0/3651328630 >> [v2:172.21.15.16:6800/1678074233,v1:172.21.15.16:6801/1678074233] conn(0x7f9e50034860 msgr2=0x7f9e50036d10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-18T02:52:24.392 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:23.791+0000 7f9e6705c640 1 --2- 172.21.15.16:0/3651328630 >> [v2:172.21.15.16:6800/1678074233,v1:172.21.15.16:6801/1678074233] conn(0x7f9e50034860 0x7f9e50036d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:24.392 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:23.791+0000 7f9e6705c640 1 -- 172.21.15.16:0/3651328630 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e60075ba0 msgr2=0x7f9e60110b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:24.393 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:23.791+0000 7f9e6705c640 1 --2- 172.21.15.16:0/3651328630 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e60075ba0 0x7f9e60110b10 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9e54031b90 tx=0x7f9e54008040 comp rx=0 tx=0).stop 2024-06-18T02:52:24.393 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:23.791+0000 7f9e6705c640 1 -- 172.21.15.16:0/3651328630 shutdown_connections 2024-06-18T02:52:24.393 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:23.791+0000 7f9e6705c640 1 --2- 172.21.15.16:0/3651328630 >> [v2:172.21.15.16:6800/1678074233,v1:172.21.15.16:6801/1678074233] conn(0x7f9e50034860 0x7f9e50036d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:24.393 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:23.791+0000 7f9e6705c640 1 --2- 172.21.15.16:0/3651328630 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e60075ba0 0x7f9e60110b10 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:24.393 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:23.791+0000 7f9e6705c640 1 -- 172.21.15.16:0/3651328630 >> 172.21.15.16:0/3651328630 conn(0x7f9e600fc430 msgr2=0x7f9e600fcfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:52:24.393 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:23.791+0000 7f9e6705c640 1 -- 172.21.15.16:0/3651328630 shutdown_connections 2024-06-18T02:52:24.393 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:23.792+0000 7f9e6705c640 1 -- 172.21.15.16:0/3651328630 wait complete. 2024-06-18T02:52:24.393 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for the mgr to restart... 2024-06-18T02:52:24.393 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for mgr epoch 5... 2024-06-18T02:52:24.929 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:24 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/3651328630' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-06-18T02:52:27.556 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:27 smithi016 ceph-mon[29214]: Active manager daemon smithi016.amxydz restarted 2024-06-18T02:52:27.556 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:27 smithi016 ceph-mon[29214]: Activating manager daemon smithi016.amxydz 2024-06-18T02:52:27.556 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:27 smithi016 ceph-mon[29214]: osdmap e2: 0 total, 0 up, 0 in 2024-06-18T02:52:27.556 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:27 smithi016 ceph-mon[29214]: mgrmap e6: smithi016.amxydz(active, starting, since 0.00332032s) 2024-06-18T02:52:27.556 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:27 smithi016 ceph-mon[29214]: from='mgr.14118 172.21.15.16:0/47051786' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "mon metadata", "id": "smithi016"}]: dispatch 2024-06-18T02:52:27.556 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:27 smithi016 ceph-mon[29214]: from='mgr.14118 172.21.15.16:0/47051786' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "mgr metadata", "who": "smithi016.amxydz", "id": "smithi016.amxydz"}]: dispatch 2024-06-18T02:52:27.557 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:27 smithi016 ceph-mon[29214]: from='mgr.14118 172.21.15.16:0/47051786' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-18T02:52:27.557 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:27 smithi016 ceph-mon[29214]: from='mgr.14118 172.21.15.16:0/47051786' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-18T02:52:27.557 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:27 smithi016 ceph-mon[29214]: from='mgr.14118 172.21.15.16:0/47051786' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-18T02:52:27.557 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:27 smithi016 ceph-mon[29214]: Manager daemon smithi016.amxydz is now available 2024-06-18T02:52:27.557 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:27 smithi016 ceph-mon[29214]: from='mgr.14118 172.21.15.16:0/47051786' entity='mgr.smithi016.amxydz' 2024-06-18T02:52:27.557 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:27 smithi016 ceph-mon[29214]: from='mgr.14118 172.21.15.16:0/47051786' entity='mgr.smithi016.amxydz' 2024-06-18T02:52:27.558 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:27 smithi016 ceph-mon[29214]: from='mgr.14118 172.21.15.16:0/47051786' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:52:27.558 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:27 smithi016 ceph-mon[29214]: from='mgr.14118 172.21.15.16:0/47051786' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:52:27.558 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:27 smithi016 ceph-mon[29214]: from='mgr.14118 172.21.15.16:0/47051786' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:52:27.558 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:27 smithi016 ceph-mon[29214]: from='mgr.14118 172.21.15.16:0/47051786' entity='mgr.smithi016.amxydz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi016.amxydz/mirror_snapshot_schedule"}]: dispatch 2024-06-18T02:52:27.558 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:27 smithi016 ceph-mon[29214]: from='mgr.14118 172.21.15.16:0/47051786' entity='mgr.smithi016.amxydz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi016.amxydz/trash_purge_schedule"}]: dispatch 2024-06-18T02:52:28.704 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:28 smithi016 ceph-mon[29214]: Found migration_current of "None". Setting to last migration. 2024-06-18T02:52:28.704 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:28 smithi016 ceph-mon[29214]: mgrmap e7: smithi016.amxydz(active, since 1.00523s) 2024-06-18T02:52:28.969 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-06-18T02:52:28.969 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-06-18T02:52:28.969 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "initialized": true 2024-06-18T02:52:28.969 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-06-18T02:52:28.969 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:24.784+0000 7f316bb1a640 1 Processor -- start 2024-06-18T02:52:28.969 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:24.784+0000 7f316bb1a640 1 -- start start 2024-06-18T02:52:28.969 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:24.784+0000 7f316bb1a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3164077d30 0x7f3164076260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:28.969 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:24.784+0000 7f316bb1a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3164078140 con 0x7f3164077d30 2024-06-18T02:52:28.969 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:24.784+0000 7f31698b6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3164077d30 0x7f3164076260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:28.969 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:24.784+0000 7f31698b6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3164077d30 0x7f3164076260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39288/0 (socket says 172.21.15.16:39288) 2024-06-18T02:52:28.970 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:24.784+0000 7f31698b6640 1 -- 172.21.15.16:0/1710992460 learned_addr learned my addr 172.21.15.16:0/1710992460 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:52:28.970 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:24.785+0000 7f31698b6640 1 -- 172.21.15.16:0/1710992460 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31640767a0 con 0x7f3164077d30 2024-06-18T02:52:28.970 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:24.785+0000 7f31698b6640 1 --2- 172.21.15.16:0/1710992460 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3164077d30 0x7f3164076260 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3160009840 tx=0x7f31600091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b80c0c7d5de29ec1 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:28.970 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:24.786+0000 7f31688b4640 1 -- 172.21.15.16:0/1710992460 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3160032440 con 0x7f3164077d30 2024-06-18T02:52:28.970 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:24.786+0000 7f31688b4640 1 -- 172.21.15.16:0/1710992460 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f3160032a30 con 0x7f3164077d30 2024-06-18T02:52:28.970 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:24.786+0000 7f31688b4640 1 -- 172.21.15.16:0/1710992460 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3160031ad0 con 0x7f3164077d30 2024-06-18T02:52:28.970 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:24.786+0000 7f316bb1a640 1 -- 172.21.15.16:0/1710992460 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3164077d30 msgr2=0x7f3164076260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:28.970 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:24.786+0000 7f316bb1a640 1 --2- 172.21.15.16:0/1710992460 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3164077d30 0x7f3164076260 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3160009840 tx=0x7f31600091d0 comp rx=0 tx=0).stop 2024-06-18T02:52:28.970 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:24.786+0000 7f316bb1a640 1 -- 172.21.15.16:0/1710992460 shutdown_connections 2024-06-18T02:52:28.970 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:24.786+0000 7f316bb1a640 1 --2- 172.21.15.16:0/1710992460 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3164077d30 0x7f3164076260 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:28.970 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:24.786+0000 7f316bb1a640 1 -- 172.21.15.16:0/1710992460 >> 172.21.15.16:0/1710992460 conn(0x7f31640fd470 msgr2=0x7f31640ff880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:52:28.970 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:24.786+0000 7f316bb1a640 1 -- 172.21.15.16:0/1710992460 shutdown_connections 2024-06-18T02:52:28.971 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:24.787+0000 7f316bb1a640 1 -- 172.21.15.16:0/1710992460 wait complete. 2024-06-18T02:52:28.971 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:24.787+0000 7f316bb1a640 1 Processor -- start 2024-06-18T02:52:28.971 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:24.787+0000 7f316bb1a640 1 -- start start 2024-06-18T02:52:28.971 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:24.787+0000 7f316bb1a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3164077d30 0x7f3164112ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:28.971 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:24.787+0000 7f316bb1a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3164113220 con 0x7f3164077d30 2024-06-18T02:52:28.971 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:24.787+0000 7f31698b6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3164077d30 0x7f3164112ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:28.971 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:24.787+0000 7f31698b6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3164077d30 0x7f3164112ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39294/0 (socket says 172.21.15.16:39294) 2024-06-18T02:52:28.971 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:24.787+0000 7f31698b6640 1 -- 172.21.15.16:0/334455306 learned_addr learned my addr 172.21.15.16:0/334455306 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:52:28.971 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:24.788+0000 7f31698b6640 1 -- 172.21.15.16:0/334455306 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3160009450 con 0x7f3164077d30 2024-06-18T02:52:28.971 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:24.788+0000 7f31698b6640 1 --2- 172.21.15.16:0/334455306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3164077d30 0x7f3164112ce0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3160031b90 tx=0x7f3160008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:28.971 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:24.788+0000 7f315a7fc640 1 -- 172.21.15.16:0/334455306 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f31600328b0 con 0x7f3164077d30 2024-06-18T02:52:28.971 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:24.788+0000 7f315a7fc640 1 -- 172.21.15.16:0/334455306 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f316003d8e0 con 0x7f3164077d30 2024-06-18T02:52:28.971 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:24.788+0000 7f316bb1a640 1 -- 172.21.15.16:0/334455306 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3164113420 con 0x7f3164077d30 2024-06-18T02:52:28.972 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:24.788+0000 7f315a7fc640 1 -- 172.21.15.16:0/334455306 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f316003ba20 con 0x7f3164077d30 2024-06-18T02:52:28.972 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:24.788+0000 7f316bb1a640 1 -- 172.21.15.16:0/334455306 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3164113890 con 0x7f3164077d30 2024-06-18T02:52:28.972 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:24.789+0000 7f315a7fc640 1 -- 172.21.15.16:0/334455306 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f3160039070 con 0x7f3164077d30 2024-06-18T02:52:28.972 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:24.789+0000 7f315a7fc640 1 --2- 172.21.15.16:0/334455306 >> [v2:172.21.15.16:6800/1678074233,v1:172.21.15.16:6801/1678074233] conn(0x7f3150034860 0x7f3150036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:28.972 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:24.789+0000 7f316bb1a640 1 -- 172.21.15.16:0/334455306 --> [v2:172.21.15.16:6800/1678074233,v1:172.21.15.16:6801/1678074233] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f316405ead0 con 0x7f3150034860 2024-06-18T02:52:28.972 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:24.789+0000 7f315a7fc640 1 -- 172.21.15.16:0/334455306 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f31600672c0 con 0x7f3164077d30 2024-06-18T02:52:28.972 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:24.789+0000 7f31690b5640 1 -- 172.21.15.16:0/334455306 >> [v2:172.21.15.16:6800/1678074233,v1:172.21.15.16:6801/1678074233] conn(0x7f3150034860 msgr2=0x7f3150036d10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/1678074233 2024-06-18T02:52:28.972 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:24.789+0000 7f31690b5640 1 --2- 172.21.15.16:0/334455306 >> [v2:172.21.15.16:6800/1678074233,v1:172.21.15.16:6801/1678074233] conn(0x7f3150034860 0x7f3150036d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-18T02:52:28.972 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:24.990+0000 7f31690b5640 1 -- 172.21.15.16:0/334455306 >> [v2:172.21.15.16:6800/1678074233,v1:172.21.15.16:6801/1678074233] conn(0x7f3150034860 msgr2=0x7f3150036d10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/1678074233 2024-06-18T02:52:28.972 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:24.990+0000 7f31690b5640 1 --2- 172.21.15.16:0/334455306 >> [v2:172.21.15.16:6800/1678074233,v1:172.21.15.16:6801/1678074233] conn(0x7f3150034860 0x7f3150036d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-18T02:52:28.972 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:25.391+0000 7f31690b5640 1 -- 172.21.15.16:0/334455306 >> [v2:172.21.15.16:6800/1678074233,v1:172.21.15.16:6801/1678074233] conn(0x7f3150034860 msgr2=0x7f3150036d10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/1678074233 2024-06-18T02:52:28.972 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:25.391+0000 7f31690b5640 1 --2- 172.21.15.16:0/334455306 >> [v2:172.21.15.16:6800/1678074233,v1:172.21.15.16:6801/1678074233] conn(0x7f3150034860 0x7f3150036d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-18T02:52:28.973 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:26.192+0000 7f31690b5640 1 -- 172.21.15.16:0/334455306 >> [v2:172.21.15.16:6800/1678074233,v1:172.21.15.16:6801/1678074233] conn(0x7f3150034860 msgr2=0x7f3150036d10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/1678074233 2024-06-18T02:52:28.973 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:26.192+0000 7f31690b5640 1 --2- 172.21.15.16:0/334455306 >> [v2:172.21.15.16:6800/1678074233,v1:172.21.15.16:6801/1678074233] conn(0x7f3150034860 0x7f3150036d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-18T02:52:28.973 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:27.325+0000 7f315a7fc640 1 -- 172.21.15.16:0/334455306 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mgrmap(e 6) v1 ==== 41713+0+0 (secure 0 0 0) 0x7f316003bc40 con 0x7f3164077d30 2024-06-18T02:52:28.973 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:27.326+0000 7f315a7fc640 1 -- 172.21.15.16:0/334455306 >> [v2:172.21.15.16:6800/1678074233,v1:172.21.15.16:6801/1678074233] conn(0x7f3150034860 msgr2=0x7f3150036d10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-18T02:52:28.973 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:27.326+0000 7f315a7fc640 1 --2- 172.21.15.16:0/334455306 >> [v2:172.21.15.16:6800/1678074233,v1:172.21.15.16:6801/1678074233] conn(0x7f3150034860 0x7f3150036d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:28.973 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:28.326+0000 7f315a7fc640 1 -- 172.21.15.16:0/334455306 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 7) v1 ==== 41840+0+0 (secure 0 0 0) 0x7f3160046540 con 0x7f3164077d30 2024-06-18T02:52:28.973 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:28.327+0000 7f315a7fc640 1 --2- 172.21.15.16:0/334455306 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7f3150037fd0 0x7f315003a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:28.973 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:28.327+0000 7f315a7fc640 1 -- 172.21.15.16:0/334455306 --> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f316405ead0 con 0x7f3150037fd0 2024-06-18T02:52:28.973 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:28.327+0000 7f31690b5640 1 --2- 172.21.15.16:0/334455306 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7f3150037fd0 0x7f315003a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:28.973 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:28.327+0000 7f31690b5640 1 --2- 172.21.15.16:0/334455306 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7f3150037fd0 0x7f315003a3b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f3154003940 tx=0x7f3154009350 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:28.973 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:28.328+0000 7f315a7fc640 1 -- 172.21.15.16:0/334455306 <== mgr.14118 v2:172.21.15.16:6800/1807632468 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7f316405ead0 con 0x7f3150037fd0 2024-06-18T02:52:28.973 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:28.333+0000 7f316bb1a640 1 -- 172.21.15.16:0/334455306 --> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f316404b480 con 0x7f3150037fd0 2024-06-18T02:52:28.973 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:28.334+0000 7f315a7fc640 1 -- 172.21.15.16:0/334455306 <== mgr.14118 v2:172.21.15.16:6800/1807632468 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f316404b480 con 0x7f3150037fd0 2024-06-18T02:52:28.974 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:28.334+0000 7f316bb1a640 1 -- 172.21.15.16:0/334455306 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7f3150037fd0 msgr2=0x7f315003a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:28.974 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:28.334+0000 7f316bb1a640 1 --2- 172.21.15.16:0/334455306 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7f3150037fd0 0x7f315003a3b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f3154003940 tx=0x7f3154009350 comp rx=0 tx=0).stop 2024-06-18T02:52:28.974 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:28.334+0000 7f316bb1a640 1 -- 172.21.15.16:0/334455306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3164077d30 msgr2=0x7f3164112ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:28.974 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:28.334+0000 7f316bb1a640 1 --2- 172.21.15.16:0/334455306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3164077d30 0x7f3164112ce0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3160031b90 tx=0x7f3160008040 comp rx=0 tx=0).stop 2024-06-18T02:52:28.974 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:28.334+0000 7f316bb1a640 1 -- 172.21.15.16:0/334455306 shutdown_connections 2024-06-18T02:52:28.974 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:28.334+0000 7f316bb1a640 1 --2- 172.21.15.16:0/334455306 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7f3150037fd0 0x7f315003a3b0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:28.974 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:28.334+0000 7f316bb1a640 1 --2- 172.21.15.16:0/334455306 >> [v2:172.21.15.16:6800/1678074233,v1:172.21.15.16:6801/1678074233] conn(0x7f3150034860 0x7f3150036d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:28.974 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:28.334+0000 7f316bb1a640 1 --2- 172.21.15.16:0/334455306 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3164077d30 0x7f3164112ce0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:28.974 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:28.334+0000 7f316bb1a640 1 -- 172.21.15.16:0/334455306 >> 172.21.15.16:0/334455306 conn(0x7f31640fd470 msgr2=0x7f31640fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:52:28.974 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:28.334+0000 7f316bb1a640 1 -- 172.21.15.16:0/334455306 shutdown_connections 2024-06-18T02:52:28.974 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:28.334+0000 7f316bb1a640 1 -- 172.21.15.16:0/334455306 wait complete. 2024-06-18T02:52:28.974 INFO:teuthology.orchestra.run.smithi016.stdout:mgr epoch 5 is available 2024-06-18T02:52:28.974 INFO:teuthology.orchestra.run.smithi016.stdout:Setting orchestrator backend to cephadm... 2024-06-18T02:52:29.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:29 smithi016 ceph-mon[29214]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-06-18T02:52:29.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:29 smithi016 ceph-mon[29214]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-06-18T02:52:30.191 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:29.365+0000 7fd3cf00e640 1 Processor -- start 2024-06-18T02:52:30.192 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:29.365+0000 7fd3cf00e640 1 -- start start 2024-06-18T02:52:30.192 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:29.365+0000 7fd3cf00e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3c8075ba0 0x7fd3c8104940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:30.192 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:29.365+0000 7fd3cf00e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3c8104e80 con 0x7fd3c8075ba0 2024-06-18T02:52:30.192 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:29.365+0000 7fd3ccdaa640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3c8075ba0 0x7fd3c8104940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:30.192 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:29.365+0000 7fd3ccdaa640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3c8075ba0 0x7fd3c8104940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39360/0 (socket says 172.21.15.16:39360) 2024-06-18T02:52:30.192 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:29.365+0000 7fd3ccdaa640 1 -- 172.21.15.16:0/664379246 learned_addr learned my addr 172.21.15.16:0/664379246 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:52:30.192 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:29.366+0000 7fd3ccdaa640 1 -- 172.21.15.16:0/664379246 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3c8104fc0 con 0x7fd3c8075ba0 2024-06-18T02:52:30.192 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:29.366+0000 7fd3ccdaa640 1 --2- 172.21.15.16:0/664379246 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3c8075ba0 0x7fd3c8104940 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd3bc009840 tx=0x7fd3bc0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=505dfaf852990e5e server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:30.192 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:29.367+0000 7fd3c77fe640 1 -- 172.21.15.16:0/664379246 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd3bc032440 con 0x7fd3c8075ba0 2024-06-18T02:52:30.192 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:29.367+0000 7fd3c77fe640 1 -- 172.21.15.16:0/664379246 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fd3bc032a30 con 0x7fd3c8075ba0 2024-06-18T02:52:30.192 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:29.367+0000 7fd3c77fe640 1 -- 172.21.15.16:0/664379246 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd3bc031ad0 con 0x7fd3c8075ba0 2024-06-18T02:52:30.192 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:29.367+0000 7fd3cf00e640 1 -- 172.21.15.16:0/664379246 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3c8075ba0 msgr2=0x7fd3c8104940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:30.193 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:29.367+0000 7fd3cf00e640 1 --2- 172.21.15.16:0/664379246 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3c8075ba0 0x7fd3c8104940 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd3bc009840 tx=0x7fd3bc0091d0 comp rx=0 tx=0).stop 2024-06-18T02:52:30.193 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:29.367+0000 7fd3cf00e640 1 -- 172.21.15.16:0/664379246 shutdown_connections 2024-06-18T02:52:30.193 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:29.367+0000 7fd3cf00e640 1 --2- 172.21.15.16:0/664379246 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3c8075ba0 0x7fd3c8104940 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:30.193 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:29.367+0000 7fd3cf00e640 1 -- 172.21.15.16:0/664379246 >> 172.21.15.16:0/664379246 conn(0x7fd3c80fc430 msgr2=0x7fd3c80fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:52:30.193 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:29.367+0000 7fd3cf00e640 1 -- 172.21.15.16:0/664379246 shutdown_connections 2024-06-18T02:52:30.193 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:29.367+0000 7fd3cf00e640 1 -- 172.21.15.16:0/664379246 wait complete. 2024-06-18T02:52:30.193 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:29.368+0000 7fd3cf00e640 1 Processor -- start 2024-06-18T02:52:30.193 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:29.368+0000 7fd3cf00e640 1 -- start start 2024-06-18T02:52:30.193 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:29.368+0000 7fd3cf00e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3c8075ba0 0x7fd3c8110b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:30.193 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:29.368+0000 7fd3cf00e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3c8111050 con 0x7fd3c8075ba0 2024-06-18T02:52:30.193 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:29.368+0000 7fd3ccdaa640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3c8075ba0 0x7fd3c8110b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:30.193 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:29.368+0000 7fd3ccdaa640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3c8075ba0 0x7fd3c8110b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39370/0 (socket says 172.21.15.16:39370) 2024-06-18T02:52:30.193 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:29.368+0000 7fd3ccdaa640 1 -- 172.21.15.16:0/3554190086 learned_addr learned my addr 172.21.15.16:0/3554190086 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:52:30.194 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:29.369+0000 7fd3ccdaa640 1 -- 172.21.15.16:0/3554190086 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3bc009450 con 0x7fd3c8075ba0 2024-06-18T02:52:30.194 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:29.369+0000 7fd3ccdaa640 1 --2- 172.21.15.16:0/3554190086 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3c8075ba0 0x7fd3c8110b10 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd3bc006f00 tx=0x7fd3bc008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:30.194 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:29.369+0000 7fd3c57fa640 1 -- 172.21.15.16:0/3554190086 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd3bc03d450 con 0x7fd3c8075ba0 2024-06-18T02:52:30.194 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:29.369+0000 7fd3c57fa640 1 -- 172.21.15.16:0/3554190086 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fd3bc03da40 con 0x7fd3c8075ba0 2024-06-18T02:52:30.194 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:29.369+0000 7fd3cf00e640 1 -- 172.21.15.16:0/3554190086 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3c8111250 con 0x7fd3c8075ba0 2024-06-18T02:52:30.194 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:29.369+0000 7fd3c57fa640 1 -- 172.21.15.16:0/3554190086 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd3bc03bb10 con 0x7fd3c8075ba0 2024-06-18T02:52:30.194 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:29.369+0000 7fd3cf00e640 1 -- 172.21.15.16:0/3554190086 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3c81116c0 con 0x7fd3c8075ba0 2024-06-18T02:52:30.194 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:29.370+0000 7fd3c57fa640 1 -- 172.21.15.16:0/3554190086 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 7) v1 ==== 41840+0+0 (secure 0 0 0) 0x7fd3bc039070 con 0x7fd3c8075ba0 2024-06-18T02:52:30.194 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:29.370+0000 7fd3c57fa640 1 --2- 172.21.15.16:0/3554190086 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7fd3b80346e0 0x7fd3b8036b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:30.194 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:29.370+0000 7fd3cf00e640 1 -- 172.21.15.16:0/3554190086 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3c804b480 con 0x7fd3c8075ba0 2024-06-18T02:52:30.194 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:29.370+0000 7fd3c57fa640 1 -- 172.21.15.16:0/3554190086 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fd3bc067570 con 0x7fd3c8075ba0 2024-06-18T02:52:30.194 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:29.370+0000 7fd3c7fff640 1 --2- 172.21.15.16:0/3554190086 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7fd3b80346e0 0x7fd3b8036b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:30.195 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:29.371+0000 7fd3c7fff640 1 --2- 172.21.15.16:0/3554190086 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7fd3b80346e0 0x7fd3b8036b90 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd3b4006fd0 tx=0x7fd3b4006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:30.195 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:29.375+0000 7fd3c57fa640 1 -- 172.21.15.16:0/3554190086 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fd3bc03bdb0 con 0x7fd3c8075ba0 2024-06-18T02:52:30.195 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:29.377+0000 7fd3c57fa640 1 -- 172.21.15.16:0/3554190086 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7fd3bc046540 con 0x7fd3c8075ba0 2024-06-18T02:52:30.195 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:29.562+0000 7fd3cf00e640 1 -- 172.21.15.16:0/3554190086 --> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7fd3c81116c0 con 0x7fd3b80346e0 2024-06-18T02:52:30.195 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:29.570+0000 7fd3c57fa640 1 -- 172.21.15.16:0/3554190086 <== mgr.14118 v2:172.21.15.16:6800/1807632468 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fd3c81116c0 con 0x7fd3b80346e0 2024-06-18T02:52:30.195 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:29.572+0000 7fd3cf00e640 1 -- 172.21.15.16:0/3554190086 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7fd3b80346e0 msgr2=0x7fd3b8036b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:30.195 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:29.572+0000 7fd3cf00e640 1 --2- 172.21.15.16:0/3554190086 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7fd3b80346e0 0x7fd3b8036b90 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd3b4006fd0 tx=0x7fd3b4006db0 comp rx=0 tx=0).stop 2024-06-18T02:52:30.195 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:29.572+0000 7fd3cf00e640 1 -- 172.21.15.16:0/3554190086 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3c8075ba0 msgr2=0x7fd3c8110b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:30.195 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:29.572+0000 7fd3cf00e640 1 --2- 172.21.15.16:0/3554190086 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3c8075ba0 0x7fd3c8110b10 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd3bc006f00 tx=0x7fd3bc008040 comp rx=0 tx=0).stop 2024-06-18T02:52:30.195 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:29.572+0000 7fd3cf00e640 1 -- 172.21.15.16:0/3554190086 shutdown_connections 2024-06-18T02:52:30.195 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:29.572+0000 7fd3cf00e640 1 --2- 172.21.15.16:0/3554190086 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7fd3b80346e0 0x7fd3b8036b90 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:30.195 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:29.572+0000 7fd3cf00e640 1 --2- 172.21.15.16:0/3554190086 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd3c8075ba0 0x7fd3c8110b10 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:30.195 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:29.572+0000 7fd3cf00e640 1 -- 172.21.15.16:0/3554190086 >> 172.21.15.16:0/3554190086 conn(0x7fd3c80fc430 msgr2=0x7fd3c80fcfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:52:30.196 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:29.572+0000 7fd3cf00e640 1 -- 172.21.15.16:0/3554190086 shutdown_connections 2024-06-18T02:52:30.196 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:29.572+0000 7fd3cf00e640 1 -- 172.21.15.16:0/3554190086 wait complete. 2024-06-18T02:52:30.486 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:30 smithi016 ceph-mon[29214]: mgrmap e8: smithi016.amxydz(active, since 2s) 2024-06-18T02:52:30.487 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:30 smithi016 ceph-mon[29214]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-06-18T02:52:30.487 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:30 smithi016 ceph-mon[29214]: from='mgr.14118 172.21.15.16:0/47051786' entity='mgr.smithi016.amxydz' 2024-06-18T02:52:30.487 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:30 smithi016 ceph-mon[29214]: from='mgr.14118 172.21.15.16:0/47051786' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:52:30.487 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:30 smithi016 ceph-mon[29214]: [18/Jun/2024:02:52:29] ENGINE Bus STARTING 2024-06-18T02:52:30.487 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:30 smithi016 ceph-mon[29214]: from='mgr.14118 172.21.15.16:0/47051786' entity='mgr.smithi016.amxydz' 2024-06-18T02:52:30.487 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:30 smithi016 ceph-mon[29214]: from='mgr.14118 172.21.15.16:0/47051786' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:52:31.289 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout value unchanged 2024-06-18T02:52:31.289 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:30.574+0000 7fc28a452640 1 Processor -- start 2024-06-18T02:52:31.289 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:30.574+0000 7fc28a452640 1 -- start start 2024-06-18T02:52:31.290 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:30.574+0000 7fc28a452640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc284101290 0x7fc2841016a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:31.290 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:30.575+0000 7fc28a452640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc284101be0 con 0x7fc284101290 2024-06-18T02:52:31.290 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:30.575+0000 7fc283fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc284101290 0x7fc2841016a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:31.290 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:30.575+0000 7fc283fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc284101290 0x7fc2841016a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39374/0 (socket says 172.21.15.16:39374) 2024-06-18T02:52:31.290 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:30.575+0000 7fc283fff640 1 -- 172.21.15.16:0/3648520471 learned_addr learned my addr 172.21.15.16:0/3648520471 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:52:31.290 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:30.575+0000 7fc283fff640 1 -- 172.21.15.16:0/3648520471 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc284101d20 con 0x7fc284101290 2024-06-18T02:52:31.290 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:30.576+0000 7fc283fff640 1 --2- 172.21.15.16:0/3648520471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc284101290 0x7fc2841016a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc274009840 tx=0x7fc2740091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c20fba23a4cf504f server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:31.290 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:30.576+0000 7fc282ffd640 1 -- 172.21.15.16:0/3648520471 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc274032440 con 0x7fc284101290 2024-06-18T02:52:31.290 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:30.576+0000 7fc282ffd640 1 -- 172.21.15.16:0/3648520471 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fc274032a30 con 0x7fc284101290 2024-06-18T02:52:31.290 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:30.576+0000 7fc282ffd640 1 -- 172.21.15.16:0/3648520471 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc274031ad0 con 0x7fc284101290 2024-06-18T02:52:31.290 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:30.576+0000 7fc28a452640 1 -- 172.21.15.16:0/3648520471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc284101290 msgr2=0x7fc2841016a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:31.290 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:30.576+0000 7fc28a452640 1 --2- 172.21.15.16:0/3648520471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc284101290 0x7fc2841016a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc274009840 tx=0x7fc2740091d0 comp rx=0 tx=0).stop 2024-06-18T02:52:31.290 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:30.577+0000 7fc28a452640 1 -- 172.21.15.16:0/3648520471 shutdown_connections 2024-06-18T02:52:31.291 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:30.577+0000 7fc28a452640 1 --2- 172.21.15.16:0/3648520471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc284101290 0x7fc2841016a0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:31.291 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:30.577+0000 7fc28a452640 1 -- 172.21.15.16:0/3648520471 >> 172.21.15.16:0/3648520471 conn(0x7fc2840fbbc0 msgr2=0x7fc2840fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:52:31.291 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:30.577+0000 7fc28a452640 1 -- 172.21.15.16:0/3648520471 shutdown_connections 2024-06-18T02:52:31.291 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:30.577+0000 7fc28a452640 1 -- 172.21.15.16:0/3648520471 wait complete. 2024-06-18T02:52:31.291 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:30.577+0000 7fc28a452640 1 Processor -- start 2024-06-18T02:52:31.291 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:30.577+0000 7fc28a452640 1 -- start start 2024-06-18T02:52:31.291 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:30.578+0000 7fc28a452640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc284101290 0x7fc284100af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:31.291 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:30.578+0000 7fc28a452640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc284113770 con 0x7fc284101290 2024-06-18T02:52:31.291 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:30.578+0000 7fc283fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc284101290 0x7fc284100af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:31.291 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:30.578+0000 7fc283fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc284101290 0x7fc284100af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39390/0 (socket says 172.21.15.16:39390) 2024-06-18T02:52:31.291 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:30.578+0000 7fc283fff640 1 -- 172.21.15.16:0/1103632092 learned_addr learned my addr 172.21.15.16:0/1103632092 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:52:31.291 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:30.578+0000 7fc283fff640 1 -- 172.21.15.16:0/1103632092 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc274009450 con 0x7fc284101290 2024-06-18T02:52:31.291 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:30.578+0000 7fc283fff640 1 --2- 172.21.15.16:0/1103632092 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc284101290 0x7fc284100af0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fc274009420 tx=0x7fc274008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:31.291 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:30.578+0000 7fc280ff9640 1 -- 172.21.15.16:0/1103632092 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc27403d5b0 con 0x7fc284101290 2024-06-18T02:52:31.292 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:30.578+0000 7fc280ff9640 1 -- 172.21.15.16:0/1103632092 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fc27403dba0 con 0x7fc284101290 2024-06-18T02:52:31.292 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:30.578+0000 7fc280ff9640 1 -- 172.21.15.16:0/1103632092 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc27403bb40 con 0x7fc284101290 2024-06-18T02:52:31.292 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:30.579+0000 7fc28a452640 1 -- 172.21.15.16:0/1103632092 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2841138b0 con 0x7fc284101290 2024-06-18T02:52:31.292 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:30.579+0000 7fc28a452640 1 -- 172.21.15.16:0/1103632092 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc284113d20 con 0x7fc284101290 2024-06-18T02:52:31.292 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:30.579+0000 7fc280ff9640 1 -- 172.21.15.16:0/1103632092 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7fc274039070 con 0x7fc284101290 2024-06-18T02:52:31.292 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:30.580+0000 7fc280ff9640 1 --2- 172.21.15.16:0/1103632092 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7fc270034810 0x7fc270036cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:31.292 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:30.580+0000 7fc280ff9640 1 -- 172.21.15.16:0/1103632092 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fc274067810 con 0x7fc284101290 2024-06-18T02:52:31.292 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:30.580+0000 7fc28a452640 1 -- 172.21.15.16:0/1103632092 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc28404b480 con 0x7fc284101290 2024-06-18T02:52:31.292 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:30.580+0000 7fc2837fe640 1 --2- 172.21.15.16:0/1103632092 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7fc270034810 0x7fc270036cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:31.292 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:30.580+0000 7fc2837fe640 1 --2- 172.21.15.16:0/1103632092 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7fc270034810 0x7fc270036cc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc26c006fd0 tx=0x7fc26c006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:31.292 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:30.585+0000 7fc280ff9640 1 -- 172.21.15.16:0/1103632092 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fc274041080 con 0x7fc284101290 2024-06-18T02:52:31.292 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:30.760+0000 7fc28a452640 1 -- 172.21.15.16:0/1103632092 --> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7fc2840fce30 con 0x7fc270034810 2024-06-18T02:52:31.292 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:30.761+0000 7fc280ff9640 1 -- 172.21.15.16:0/1103632092 <== mgr.14118 v2:172.21.15.16:6800/1807632468 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7fc2840fce30 con 0x7fc270034810 2024-06-18T02:52:31.293 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:30.763+0000 7fc28a452640 1 -- 172.21.15.16:0/1103632092 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7fc270034810 msgr2=0x7fc270036cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:31.293 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:30.763+0000 7fc28a452640 1 --2- 172.21.15.16:0/1103632092 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7fc270034810 0x7fc270036cc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc26c006fd0 tx=0x7fc26c006db0 comp rx=0 tx=0).stop 2024-06-18T02:52:31.293 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:30.763+0000 7fc28a452640 1 -- 172.21.15.16:0/1103632092 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc284101290 msgr2=0x7fc284100af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:31.293 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:30.763+0000 7fc28a452640 1 --2- 172.21.15.16:0/1103632092 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc284101290 0x7fc284100af0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fc274009420 tx=0x7fc274008040 comp rx=0 tx=0).stop 2024-06-18T02:52:31.293 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:30.763+0000 7fc28a452640 1 -- 172.21.15.16:0/1103632092 shutdown_connections 2024-06-18T02:52:31.293 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:30.763+0000 7fc28a452640 1 --2- 172.21.15.16:0/1103632092 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7fc270034810 0x7fc270036cc0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:31.293 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:30.763+0000 7fc28a452640 1 --2- 172.21.15.16:0/1103632092 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc284101290 0x7fc284100af0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:31.293 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:30.763+0000 7fc28a452640 1 -- 172.21.15.16:0/1103632092 >> 172.21.15.16:0/1103632092 conn(0x7fc2840fbbc0 msgr2=0x7fc2840fc720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:52:31.293 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:30.763+0000 7fc28a452640 1 -- 172.21.15.16:0/1103632092 shutdown_connections 2024-06-18T02:52:31.293 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:30.763+0000 7fc28a452640 1 -- 172.21.15.16:0/1103632092 wait complete. 2024-06-18T02:52:31.293 INFO:teuthology.orchestra.run.smithi016.stdout:Generating ssh key... 2024-06-18T02:52:31.651 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:31 smithi016 ceph-mon[29214]: [18/Jun/2024:02:52:29] ENGINE Serving on https://172.21.15.16:7150 2024-06-18T02:52:31.651 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:31 smithi016 ceph-mon[29214]: [18/Jun/2024:02:52:29] ENGINE Bus STARTED 2024-06-18T02:52:31.651 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:31 smithi016 ceph-mon[29214]: [18/Jun/2024:02:52:29] ENGINE Error in HTTPServer.serve 2024-06-18T02:52:31.651 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: Traceback (most recent call last): 2024-06-18T02:52:31.651 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-06-18T02:52:31.651 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: self._connections.run(self.expiration_interval) 2024-06-18T02:52:31.651 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-06-18T02:52:31.651 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: self._run(expiration_interval) 2024-06-18T02:52:31.652 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-06-18T02:52:31.652 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-06-18T02:52:31.652 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-06-18T02:52:31.652 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-06-18T02:52:31.652 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-06-18T02:52:31.652 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: s = self.context.wrap_socket( 2024-06-18T02:52:31.652 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-06-18T02:52:31.652 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: return self.sslsocket_class._create( 2024-06-18T02:52:31.652 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-06-18T02:52:31.652 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: self.do_handshake() 2024-06-18T02:52:31.652 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-06-18T02:52:31.652 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: self._sslobj.do_handshake() 2024-06-18T02:52:31.652 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1129) 2024-06-18T02:52:32.720 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:32 smithi016 ceph-mon[29214]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-06-18T02:52:33.154 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:31.662+0000 7f1d5a575640 1 Processor -- start 2024-06-18T02:52:33.154 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:31.662+0000 7f1d5a575640 1 -- start start 2024-06-18T02:52:33.154 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:31.663+0000 7f1d5a575640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d54077c00 0x7f1d54076220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:33.154 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:31.663+0000 7f1d5a575640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d54078010 con 0x7f1d54077c00 2024-06-18T02:52:33.154 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:31.663+0000 7f1d53fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d54077c00 0x7f1d54076220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:33.155 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:31.663+0000 7f1d53fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d54077c00 0x7f1d54076220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54126/0 (socket says 172.21.15.16:54126) 2024-06-18T02:52:33.155 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:31.663+0000 7f1d53fff640 1 -- 172.21.15.16:0/2897727977 learned_addr learned my addr 172.21.15.16:0/2897727977 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:52:33.155 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:31.663+0000 7f1d53fff640 1 -- 172.21.15.16:0/2897727977 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d54076760 con 0x7f1d54077c00 2024-06-18T02:52:33.155 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:31.664+0000 7f1d53fff640 1 --2- 172.21.15.16:0/2897727977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d54077c00 0x7f1d54076220 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1d44009840 tx=0x7f1d440091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1fc870f8c1eac54 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:33.155 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:31.664+0000 7f1d52ffd640 1 -- 172.21.15.16:0/2897727977 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d44032440 con 0x7f1d54077c00 2024-06-18T02:52:33.155 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:31.664+0000 7f1d52ffd640 1 -- 172.21.15.16:0/2897727977 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f1d44032a30 con 0x7f1d54077c00 2024-06-18T02:52:33.155 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:31.664+0000 7f1d52ffd640 1 -- 172.21.15.16:0/2897727977 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d44031ad0 con 0x7f1d54077c00 2024-06-18T02:52:33.155 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:31.665+0000 7f1d5a575640 1 -- 172.21.15.16:0/2897727977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d54077c00 msgr2=0x7f1d54076220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:33.155 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:31.665+0000 7f1d5a575640 1 --2- 172.21.15.16:0/2897727977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d54077c00 0x7f1d54076220 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1d44009840 tx=0x7f1d440091d0 comp rx=0 tx=0).stop 2024-06-18T02:52:33.155 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:31.665+0000 7f1d5a575640 1 -- 172.21.15.16:0/2897727977 shutdown_connections 2024-06-18T02:52:33.156 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:31.665+0000 7f1d5a575640 1 --2- 172.21.15.16:0/2897727977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d54077c00 0x7f1d54076220 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:33.156 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:31.665+0000 7f1d5a575640 1 -- 172.21.15.16:0/2897727977 >> 172.21.15.16:0/2897727977 conn(0x7f1d540fd5b0 msgr2=0x7f1d540ff9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:52:33.156 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:31.665+0000 7f1d5a575640 1 -- 172.21.15.16:0/2897727977 shutdown_connections 2024-06-18T02:52:33.156 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:31.665+0000 7f1d5a575640 1 -- 172.21.15.16:0/2897727977 wait complete. 2024-06-18T02:52:33.156 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:31.665+0000 7f1d5a575640 1 Processor -- start 2024-06-18T02:52:33.156 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:31.666+0000 7f1d5a575640 1 -- start start 2024-06-18T02:52:33.156 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:31.666+0000 7f1d5a575640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d54077c00 0x7f1d54113020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:33.156 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:31.666+0000 7f1d5a575640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d54113560 con 0x7f1d54077c00 2024-06-18T02:52:33.156 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:31.666+0000 7f1d53fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d54077c00 0x7f1d54113020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:33.156 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:31.666+0000 7f1d53fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d54077c00 0x7f1d54113020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54132/0 (socket says 172.21.15.16:54132) 2024-06-18T02:52:33.156 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:31.666+0000 7f1d53fff640 1 -- 172.21.15.16:0/918759798 learned_addr learned my addr 172.21.15.16:0/918759798 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:52:33.156 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:31.666+0000 7f1d53fff640 1 -- 172.21.15.16:0/918759798 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d44009450 con 0x7f1d54077c00 2024-06-18T02:52:33.157 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:31.667+0000 7f1d53fff640 1 --2- 172.21.15.16:0/918759798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d54077c00 0x7f1d54113020 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1d44006f00 tx=0x7f1d44008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:33.157 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:31.667+0000 7f1d50ff9640 1 -- 172.21.15.16:0/918759798 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d4403d450 con 0x7f1d54077c00 2024-06-18T02:52:33.158 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:31.667+0000 7f1d50ff9640 1 -- 172.21.15.16:0/918759798 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f1d4403da40 con 0x7f1d54077c00 2024-06-18T02:52:33.158 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:31.667+0000 7f1d5a575640 1 -- 172.21.15.16:0/918759798 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d54113760 con 0x7f1d54077c00 2024-06-18T02:52:33.158 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:31.667+0000 7f1d50ff9640 1 -- 172.21.15.16:0/918759798 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d4403bb10 con 0x7f1d54077c00 2024-06-18T02:52:33.158 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:31.667+0000 7f1d5a575640 1 -- 172.21.15.16:0/918759798 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d54113bd0 con 0x7f1d54077c00 2024-06-18T02:52:33.158 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:31.668+0000 7f1d50ff9640 1 -- 172.21.15.16:0/918759798 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f1d44039070 con 0x7f1d54077c00 2024-06-18T02:52:33.158 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:31.668+0000 7f1d5a575640 1 -- 172.21.15.16:0/918759798 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d5404b480 con 0x7f1d54077c00 2024-06-18T02:52:33.158 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:31.668+0000 7f1d50ff9640 1 --2- 172.21.15.16:0/918759798 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7f1d3c034860 0x7f1d3c036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:33.158 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:31.668+0000 7f1d50ff9640 1 -- 172.21.15.16:0/918759798 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f1d44067670 con 0x7f1d54077c00 2024-06-18T02:52:33.158 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:31.668+0000 7f1d537fe640 1 --2- 172.21.15.16:0/918759798 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7f1d3c034860 0x7f1d3c036d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:33.158 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:31.669+0000 7f1d537fe640 1 --2- 172.21.15.16:0/918759798 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7f1d3c034860 0x7f1d3c036d10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1d48006fd0 tx=0x7f1d48006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:33.159 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:31.673+0000 7f1d50ff9640 1 -- 172.21.15.16:0/918759798 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f1d4403d5b0 con 0x7f1d54077c00 2024-06-18T02:52:33.159 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:31.848+0000 7f1d5a575640 1 -- 172.21.15.16:0/918759798 --> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f1d54113bd0 con 0x7f1d3c034860 2024-06-18T02:52:33.159 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:32.439+0000 7f1d50ff9640 1 -- 172.21.15.16:0/918759798 <== mgr.14118 v2:172.21.15.16:6800/1807632468 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f1d54113bd0 con 0x7f1d3c034860 2024-06-18T02:52:33.159 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:32.441+0000 7f1d5a575640 1 -- 172.21.15.16:0/918759798 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7f1d3c034860 msgr2=0x7f1d3c036d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:33.159 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:32.441+0000 7f1d5a575640 1 --2- 172.21.15.16:0/918759798 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7f1d3c034860 0x7f1d3c036d10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1d48006fd0 tx=0x7f1d48006db0 comp rx=0 tx=0).stop 2024-06-18T02:52:33.159 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:32.441+0000 7f1d5a575640 1 -- 172.21.15.16:0/918759798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d54077c00 msgr2=0x7f1d54113020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:33.159 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:32.441+0000 7f1d5a575640 1 --2- 172.21.15.16:0/918759798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d54077c00 0x7f1d54113020 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1d44006f00 tx=0x7f1d44008040 comp rx=0 tx=0).stop 2024-06-18T02:52:33.159 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:32.441+0000 7f1d5a575640 1 -- 172.21.15.16:0/918759798 shutdown_connections 2024-06-18T02:52:33.159 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:32.441+0000 7f1d5a575640 1 --2- 172.21.15.16:0/918759798 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7f1d3c034860 0x7f1d3c036d10 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:33.159 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:32.441+0000 7f1d5a575640 1 --2- 172.21.15.16:0/918759798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d54077c00 0x7f1d54113020 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:33.159 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:32.441+0000 7f1d5a575640 1 -- 172.21.15.16:0/918759798 >> 172.21.15.16:0/918759798 conn(0x7f1d540fd5b0 msgr2=0x7f1d540fe2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:52:33.159 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:32.441+0000 7f1d5a575640 1 -- 172.21.15.16:0/918759798 shutdown_connections 2024-06-18T02:52:33.159 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:32.441+0000 7f1d5a575640 1 -- 172.21.15.16:0/918759798 wait complete. 2024-06-18T02:52:33.673 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:33 smithi016 ceph-mon[29214]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-06-18T02:52:33.674 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:33 smithi016 ceph-mon[29214]: Generating ssh key... 2024-06-18T02:52:33.674 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:33 smithi016 ceph-mon[29214]: from='mgr.14118 172.21.15.16:0/47051786' entity='mgr.smithi016.amxydz' 2024-06-18T02:52:33.674 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:33 smithi016 ceph-mon[29214]: from='mgr.14118 172.21.15.16:0/47051786' entity='mgr.smithi016.amxydz' 2024-06-18T02:52:34.219 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCO/PgcQZoTguaJ90Ie1yUZnEfMoSQ3Di9RJYHMROBNTgFSMO8T1trbN1YsaouWWWDP5K3GjWENzPjALNRpFV543RB7W6ouU/eORcEsooGe7aNfuyNIEO55PA63GBvC9jTHUJ2T88QNUrAFguhQ3vENahhG9uT8F67WAiQXyliSDfF7rHdgzoVso72o+mmMudyIvg6sRyjbpLELSEtjPYId96jJ/huksO4xc2fDHmAYXvO2EgZNa0qWoYgJBAAU5UCdX0/gX1ePQoFuNKzHkeEIw0KiCwdhA+nrKRGCZ+Qc+zQ/DvvoDemgOVybCWwfu2Txtzd7G6ngHAKRZDc/5S8a7lxTHyAq8IVOylLsdhP3ZF2axRAlHpJMVVcuPh3UfDGmPNrK0WyavTqlOL7+F9jWaqXdAfZsWiC+f2dstgvo9yuGdLqdkWhF9bFsxon+ZnTGqUGLW+M8KEMOBfC//zEfMGQ+eeJPpfK/N5J9CzuOExVAPYPE+sSry0KPZ40OkGs= ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968 2024-06-18T02:52:34.219 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:33.488+0000 7ff49148c640 1 Processor -- start 2024-06-18T02:52:34.219 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:33.488+0000 7ff49148c640 1 -- start start 2024-06-18T02:52:34.219 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:33.488+0000 7ff49148c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff48c101cf0 0x7ff48c102100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:34.219 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:33.488+0000 7ff49148c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff48c102640 con 0x7ff48c101cf0 2024-06-18T02:52:34.219 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:33.489+0000 7ff48affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff48c101cf0 0x7ff48c102100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:34.220 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:33.489+0000 7ff48affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff48c101cf0 0x7ff48c102100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54146/0 (socket says 172.21.15.16:54146) 2024-06-18T02:52:34.220 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:33.489+0000 7ff48affd640 1 -- 172.21.15.16:0/2493914178 learned_addr learned my addr 172.21.15.16:0/2493914178 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:52:34.220 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:33.489+0000 7ff48affd640 1 -- 172.21.15.16:0/2493914178 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff48c102780 con 0x7ff48c101cf0 2024-06-18T02:52:34.220 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:33.490+0000 7ff48affd640 1 --2- 172.21.15.16:0/2493914178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff48c101cf0 0x7ff48c102100 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff47c009840 tx=0x7ff47c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6012db7eebc6d8db server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:34.220 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:33.490+0000 7ff489ffb640 1 -- 172.21.15.16:0/2493914178 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff47c032440 con 0x7ff48c101cf0 2024-06-18T02:52:34.220 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:33.490+0000 7ff489ffb640 1 -- 172.21.15.16:0/2493914178 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7ff47c032a30 con 0x7ff48c101cf0 2024-06-18T02:52:34.220 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:33.490+0000 7ff489ffb640 1 -- 172.21.15.16:0/2493914178 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff47c031ad0 con 0x7ff48c101cf0 2024-06-18T02:52:34.220 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:33.490+0000 7ff49148c640 1 -- 172.21.15.16:0/2493914178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff48c101cf0 msgr2=0x7ff48c102100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:34.220 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:33.490+0000 7ff49148c640 1 --2- 172.21.15.16:0/2493914178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff48c101cf0 0x7ff48c102100 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff47c009840 tx=0x7ff47c0091d0 comp rx=0 tx=0).stop 2024-06-18T02:52:34.220 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:33.490+0000 7ff49148c640 1 -- 172.21.15.16:0/2493914178 shutdown_connections 2024-06-18T02:52:34.220 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:33.490+0000 7ff49148c640 1 --2- 172.21.15.16:0/2493914178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff48c101cf0 0x7ff48c102100 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:34.220 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:33.490+0000 7ff49148c640 1 -- 172.21.15.16:0/2493914178 >> 172.21.15.16:0/2493914178 conn(0x7ff48c0fd4f0 msgr2=0x7ff48c0ff900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:52:34.220 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:33.490+0000 7ff49148c640 1 -- 172.21.15.16:0/2493914178 shutdown_connections 2024-06-18T02:52:34.220 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:33.490+0000 7ff49148c640 1 -- 172.21.15.16:0/2493914178 wait complete. 2024-06-18T02:52:34.221 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:33.491+0000 7ff49148c640 1 Processor -- start 2024-06-18T02:52:34.221 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:33.491+0000 7ff49148c640 1 -- start start 2024-06-18T02:52:34.221 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:33.491+0000 7ff49148c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff48c101cf0 0x7ff48c077c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:34.221 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:33.491+0000 7ff49148c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff48c078150 con 0x7ff48c101cf0 2024-06-18T02:52:34.221 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:33.491+0000 7ff48affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff48c101cf0 0x7ff48c077c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:34.221 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:33.491+0000 7ff48affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff48c101cf0 0x7ff48c077c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54152/0 (socket says 172.21.15.16:54152) 2024-06-18T02:52:34.221 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:33.491+0000 7ff48affd640 1 -- 172.21.15.16:0/205430956 learned_addr learned my addr 172.21.15.16:0/205430956 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:52:34.221 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:33.492+0000 7ff48affd640 1 -- 172.21.15.16:0/205430956 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff47c009450 con 0x7ff48c101cf0 2024-06-18T02:52:34.221 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:33.492+0000 7ff48affd640 1 --2- 172.21.15.16:0/205430956 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff48c101cf0 0x7ff48c077c10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff47c009420 tx=0x7ff47c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:34.221 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:33.492+0000 7ff4737fe640 1 -- 172.21.15.16:0/205430956 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff47c03d5b0 con 0x7ff48c101cf0 2024-06-18T02:52:34.222 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:33.492+0000 7ff49148c640 1 -- 172.21.15.16:0/205430956 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff48c0761e0 con 0x7ff48c101cf0 2024-06-18T02:52:34.222 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:33.492+0000 7ff4737fe640 1 -- 172.21.15.16:0/205430956 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7ff47c03dba0 con 0x7ff48c101cf0 2024-06-18T02:52:34.222 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:33.492+0000 7ff4737fe640 1 -- 172.21.15.16:0/205430956 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff47c03bb40 con 0x7ff48c101cf0 2024-06-18T02:52:34.222 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:33.492+0000 7ff49148c640 1 -- 172.21.15.16:0/205430956 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff48c076650 con 0x7ff48c101cf0 2024-06-18T02:52:34.222 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:33.493+0000 7ff4737fe640 1 -- 172.21.15.16:0/205430956 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7ff47c039070 con 0x7ff48c101cf0 2024-06-18T02:52:34.222 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:33.493+0000 7ff49148c640 1 -- 172.21.15.16:0/205430956 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff48c04b480 con 0x7ff48c101cf0 2024-06-18T02:52:34.222 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:33.493+0000 7ff4737fe640 1 --2- 172.21.15.16:0/205430956 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7ff474034810 0x7ff474036cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:34.222 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:33.493+0000 7ff4737fe640 1 -- 172.21.15.16:0/205430956 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7ff47c067b10 con 0x7ff48c101cf0 2024-06-18T02:52:34.222 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:33.493+0000 7ff48a7fc640 1 --2- 172.21.15.16:0/205430956 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7ff474034810 0x7ff474036cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:34.222 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:33.494+0000 7ff48a7fc640 1 --2- 172.21.15.16:0/205430956 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7ff474034810 0x7ff474036cc0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff480006fd0 tx=0x7ff480006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:34.222 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:33.498+0000 7ff4737fe640 1 -- 172.21.15.16:0/205430956 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7ff47c046400 con 0x7ff48c101cf0 2024-06-18T02:52:34.222 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:33.673+0000 7ff49148c640 1 -- 172.21.15.16:0/205430956 --> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7ff48c076cd0 con 0x7ff474034810 2024-06-18T02:52:34.222 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:33.674+0000 7ff4737fe640 1 -- 172.21.15.16:0/205430956 <== mgr.14118 v2:172.21.15.16:6800/1807632468 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7ff48c076cd0 con 0x7ff474034810 2024-06-18T02:52:34.223 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:33.675+0000 7ff49148c640 1 -- 172.21.15.16:0/205430956 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7ff474034810 msgr2=0x7ff474036cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:34.223 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:33.675+0000 7ff49148c640 1 --2- 172.21.15.16:0/205430956 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7ff474034810 0x7ff474036cc0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff480006fd0 tx=0x7ff480006db0 comp rx=0 tx=0).stop 2024-06-18T02:52:34.223 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:33.676+0000 7ff49148c640 1 -- 172.21.15.16:0/205430956 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff48c101cf0 msgr2=0x7ff48c077c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:34.223 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:33.676+0000 7ff49148c640 1 --2- 172.21.15.16:0/205430956 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff48c101cf0 0x7ff48c077c10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff47c009420 tx=0x7ff47c008040 comp rx=0 tx=0).stop 2024-06-18T02:52:34.223 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:33.676+0000 7ff49148c640 1 -- 172.21.15.16:0/205430956 shutdown_connections 2024-06-18T02:52:34.223 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:33.676+0000 7ff49148c640 1 --2- 172.21.15.16:0/205430956 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7ff474034810 0x7ff474036cc0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:34.223 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:33.676+0000 7ff49148c640 1 --2- 172.21.15.16:0/205430956 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff48c101cf0 0x7ff48c077c10 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:34.223 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:33.676+0000 7ff49148c640 1 -- 172.21.15.16:0/205430956 >> 172.21.15.16:0/205430956 conn(0x7ff48c0fd4f0 msgr2=0x7ff48c0fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:52:34.223 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:33.676+0000 7ff49148c640 1 -- 172.21.15.16:0/205430956 shutdown_connections 2024-06-18T02:52:34.223 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:33.676+0000 7ff49148c640 1 -- 172.21.15.16:0/205430956 wait complete. 2024-06-18T02:52:34.223 INFO:teuthology.orchestra.run.smithi016.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-06-18T02:52:34.223 INFO:teuthology.orchestra.run.smithi016.stdout:Adding key to root@localhost authorized_keys... 2024-06-18T02:52:34.223 INFO:teuthology.orchestra.run.smithi016.stdout:Adding host smithi016... 2024-06-18T02:52:34.462 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:34 smithi016 ceph-mon[29214]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-06-18T02:52:35.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:35 smithi016 ceph-mon[29214]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi016", "addr": "172.21.15.16", "target": ["mon-mgr", ""]}]: dispatch 2024-06-18T02:52:37.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:36 smithi016 ceph-mon[29214]: Deploying cephadm binary to smithi016 2024-06-18T02:52:38.819 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout Added host 'smithi016' with addr '172.21.15.16' 2024-06-18T02:52:38.819 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:34.620+0000 7fbd51ae7640 1 Processor -- start 2024-06-18T02:52:38.819 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:34.620+0000 7fbd51ae7640 1 -- start start 2024-06-18T02:52:38.819 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:34.620+0000 7fbd51ae7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd4c101d90 0x7fbd4c1021a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:38.819 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:34.620+0000 7fbd51ae7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd4c1026e0 con 0x7fbd4c101d90 2024-06-18T02:52:38.820 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:34.620+0000 7fbd4b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd4c101d90 0x7fbd4c1021a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:38.820 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:34.620+0000 7fbd4b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd4c101d90 0x7fbd4c1021a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54160/0 (socket says 172.21.15.16:54160) 2024-06-18T02:52:38.820 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:34.620+0000 7fbd4b7fe640 1 -- 172.21.15.16:0/3081866568 learned_addr learned my addr 172.21.15.16:0/3081866568 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:52:38.820 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:34.621+0000 7fbd4b7fe640 1 -- 172.21.15.16:0/3081866568 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd4c102820 con 0x7fbd4c101d90 2024-06-18T02:52:38.820 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:34.621+0000 7fbd4b7fe640 1 --2- 172.21.15.16:0/3081866568 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd4c101d90 0x7fbd4c1021a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c009840 tx=0x7fbd3c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e075600bf2175e32 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:38.820 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:34.622+0000 7fbd4a7fc640 1 -- 172.21.15.16:0/3081866568 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd3c032440 con 0x7fbd4c101d90 2024-06-18T02:52:38.820 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:34.622+0000 7fbd4a7fc640 1 -- 172.21.15.16:0/3081866568 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fbd3c032a30 con 0x7fbd4c101d90 2024-06-18T02:52:38.820 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:34.622+0000 7fbd4a7fc640 1 -- 172.21.15.16:0/3081866568 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd3c031ad0 con 0x7fbd4c101d90 2024-06-18T02:52:38.820 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:34.622+0000 7fbd51ae7640 1 -- 172.21.15.16:0/3081866568 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd4c101d90 msgr2=0x7fbd4c1021a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:38.820 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:34.622+0000 7fbd51ae7640 1 --2- 172.21.15.16:0/3081866568 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd4c101d90 0x7fbd4c1021a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c009840 tx=0x7fbd3c0091d0 comp rx=0 tx=0).stop 2024-06-18T02:52:38.820 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:34.622+0000 7fbd51ae7640 1 -- 172.21.15.16:0/3081866568 shutdown_connections 2024-06-18T02:52:38.820 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:34.622+0000 7fbd51ae7640 1 --2- 172.21.15.16:0/3081866568 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd4c101d90 0x7fbd4c1021a0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:38.820 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:34.622+0000 7fbd51ae7640 1 -- 172.21.15.16:0/3081866568 >> 172.21.15.16:0/3081866568 conn(0x7fbd4c0fd5b0 msgr2=0x7fbd4c0ff9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:52:38.820 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:34.622+0000 7fbd51ae7640 1 -- 172.21.15.16:0/3081866568 shutdown_connections 2024-06-18T02:52:38.821 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:34.622+0000 7fbd51ae7640 1 -- 172.21.15.16:0/3081866568 wait complete. 2024-06-18T02:52:38.821 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:34.623+0000 7fbd51ae7640 1 Processor -- start 2024-06-18T02:52:38.821 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:34.623+0000 7fbd51ae7640 1 -- start start 2024-06-18T02:52:38.821 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:34.623+0000 7fbd51ae7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd4c101d90 0x7fbd4c110e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:38.821 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:34.623+0000 7fbd51ae7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd4c1113d0 con 0x7fbd4c101d90 2024-06-18T02:52:38.821 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:34.624+0000 7fbd4b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd4c101d90 0x7fbd4c110e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:38.821 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:34.624+0000 7fbd4b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd4c101d90 0x7fbd4c110e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54172/0 (socket says 172.21.15.16:54172) 2024-06-18T02:52:38.821 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:34.624+0000 7fbd4b7fe640 1 -- 172.21.15.16:0/2659679963 learned_addr learned my addr 172.21.15.16:0/2659679963 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:52:38.821 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:34.624+0000 7fbd4b7fe640 1 -- 172.21.15.16:0/2659679963 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd3c009450 con 0x7fbd4c101d90 2024-06-18T02:52:38.821 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:34.624+0000 7fbd4b7fe640 1 --2- 172.21.15.16:0/2659679963 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd4c101d90 0x7fbd4c110e90 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c0317b0 tx=0x7fbd3c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:38.821 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:34.624+0000 7fbd33fff640 1 -- 172.21.15.16:0/2659679963 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd3c031e90 con 0x7fbd4c101d90 2024-06-18T02:52:38.821 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:34.624+0000 7fbd33fff640 1 -- 172.21.15.16:0/2659679963 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fbd3c03daf0 con 0x7fbd4c101d90 2024-06-18T02:52:38.821 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:34.624+0000 7fbd51ae7640 1 -- 172.21.15.16:0/2659679963 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd4c1115d0 con 0x7fbd4c101d90 2024-06-18T02:52:38.822 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:34.624+0000 7fbd33fff640 1 -- 172.21.15.16:0/2659679963 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd3c03baa0 con 0x7fbd4c101d90 2024-06-18T02:52:38.822 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:34.624+0000 7fbd51ae7640 1 -- 172.21.15.16:0/2659679963 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd4c111fd0 con 0x7fbd4c101d90 2024-06-18T02:52:38.822 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:34.625+0000 7fbd33fff640 1 -- 172.21.15.16:0/2659679963 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7fbd3c039070 con 0x7fbd4c101d90 2024-06-18T02:52:38.822 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:34.625+0000 7fbd51ae7640 1 -- 172.21.15.16:0/2659679963 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd4c05ead0 con 0x7fbd4c101d90 2024-06-18T02:52:38.822 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:34.625+0000 7fbd33fff640 1 --2- 172.21.15.16:0/2659679963 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7fbd34034810 0x7fbd34036cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:38.822 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:34.625+0000 7fbd33fff640 1 -- 172.21.15.16:0/2659679963 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fbd3c067ad0 con 0x7fbd4c101d90 2024-06-18T02:52:38.822 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:34.626+0000 7fbd4affd640 1 --2- 172.21.15.16:0/2659679963 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7fbd34034810 0x7fbd34036cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:38.822 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:34.626+0000 7fbd4affd640 1 --2- 172.21.15.16:0/2659679963 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7fbd34034810 0x7fbd34036cc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fbd40006fd0 tx=0x7fbd40006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:38.822 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:34.630+0000 7fbd33fff640 1 -- 172.21.15.16:0/2659679963 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fbd3c039320 con 0x7fbd4c101d90 2024-06-18T02:52:38.822 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:34.807+0000 7fbd51ae7640 1 -- 172.21.15.16:0/2659679963 --> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi016", "addr": "172.21.15.16", "target": ["mon-mgr", ""]}) v1 -- 0x7fbd4c112210 con 0x7fbd34034810 2024-06-18T02:52:38.822 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:38.288+0000 7fbd33fff640 1 -- 172.21.15.16:0/2659679963 <== mgr.14118 v2:172.21.15.16:6800/1807632468 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7fbd4c112210 con 0x7fbd34034810 2024-06-18T02:52:38.822 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:38.291+0000 7fbd51ae7640 1 -- 172.21.15.16:0/2659679963 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7fbd34034810 msgr2=0x7fbd34036cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:38.822 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:38.291+0000 7fbd51ae7640 1 --2- 172.21.15.16:0/2659679963 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7fbd34034810 0x7fbd34036cc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fbd40006fd0 tx=0x7fbd40006db0 comp rx=0 tx=0).stop 2024-06-18T02:52:38.823 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:38.291+0000 7fbd51ae7640 1 -- 172.21.15.16:0/2659679963 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd4c101d90 msgr2=0x7fbd4c110e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:38.823 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:38.291+0000 7fbd51ae7640 1 --2- 172.21.15.16:0/2659679963 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd4c101d90 0x7fbd4c110e90 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c0317b0 tx=0x7fbd3c008040 comp rx=0 tx=0).stop 2024-06-18T02:52:38.823 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:38.292+0000 7fbd51ae7640 1 -- 172.21.15.16:0/2659679963 shutdown_connections 2024-06-18T02:52:38.823 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:38.292+0000 7fbd51ae7640 1 --2- 172.21.15.16:0/2659679963 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7fbd34034810 0x7fbd34036cc0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:38.823 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:38.292+0000 7fbd51ae7640 1 --2- 172.21.15.16:0/2659679963 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd4c101d90 0x7fbd4c110e90 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:38.823 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:38.292+0000 7fbd51ae7640 1 -- 172.21.15.16:0/2659679963 >> 172.21.15.16:0/2659679963 conn(0x7fbd4c0fd5b0 msgr2=0x7fbd4c0fe2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:52:38.823 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:38.292+0000 7fbd51ae7640 1 -- 172.21.15.16:0/2659679963 shutdown_connections 2024-06-18T02:52:38.823 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:38.292+0000 7fbd51ae7640 1 -- 172.21.15.16:0/2659679963 wait complete. 2024-06-18T02:52:38.823 INFO:teuthology.orchestra.run.smithi016.stdout:Deploying mon service with default placement... 2024-06-18T02:52:39.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:39 smithi016 ceph-mon[29214]: from='mgr.14118 172.21.15.16:0/47051786' entity='mgr.smithi016.amxydz' 2024-06-18T02:52:39.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:39 smithi016 ceph-mon[29214]: Added host smithi016 2024-06-18T02:52:39.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:39 smithi016 ceph-mon[29214]: from='mgr.14118 172.21.15.16:0/47051786' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:52:40.237 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-06-18T02:52:40.238 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:39.192+0000 7faae0f0c640 1 Processor -- start 2024-06-18T02:52:40.238 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:39.192+0000 7faae0f0c640 1 -- start start 2024-06-18T02:52:40.238 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:39.193+0000 7faae0f0c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faadc1011d0 0x7faadc1015e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:40.238 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:39.193+0000 7faae0f0c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faadc101b20 con 0x7faadc1011d0 2024-06-18T02:52:40.238 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:39.193+0000 7faada59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faadc1011d0 0x7faadc1015e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:40.238 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:39.193+0000 7faada59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faadc1011d0 0x7faadc1015e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54182/0 (socket says 172.21.15.16:54182) 2024-06-18T02:52:40.238 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:39.193+0000 7faada59c640 1 -- 172.21.15.16:0/1180754280 learned_addr learned my addr 172.21.15.16:0/1180754280 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:52:40.238 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:39.193+0000 7faada59c640 1 -- 172.21.15.16:0/1180754280 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faadc101c60 con 0x7faadc1011d0 2024-06-18T02:52:40.238 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:39.194+0000 7faada59c640 1 --2- 172.21.15.16:0/1180754280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faadc1011d0 0x7faadc1015e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7faacc009840 tx=0x7faacc0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3d2f88af02dbed3 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:40.238 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:39.194+0000 7faad959a640 1 -- 172.21.15.16:0/1180754280 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faacc032440 con 0x7faadc1011d0 2024-06-18T02:52:40.238 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:39.194+0000 7faad959a640 1 -- 172.21.15.16:0/1180754280 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7faacc032a30 con 0x7faadc1011d0 2024-06-18T02:52:40.239 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:39.194+0000 7faad959a640 1 -- 172.21.15.16:0/1180754280 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faacc031ad0 con 0x7faadc1011d0 2024-06-18T02:52:40.239 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:39.195+0000 7faae0f0c640 1 -- 172.21.15.16:0/1180754280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faadc1011d0 msgr2=0x7faadc1015e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:40.239 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:39.195+0000 7faae0f0c640 1 --2- 172.21.15.16:0/1180754280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faadc1011d0 0x7faadc1015e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7faacc009840 tx=0x7faacc0091d0 comp rx=0 tx=0).stop 2024-06-18T02:52:40.239 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:39.195+0000 7faae0f0c640 1 -- 172.21.15.16:0/1180754280 shutdown_connections 2024-06-18T02:52:40.239 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:39.195+0000 7faae0f0c640 1 --2- 172.21.15.16:0/1180754280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faadc1011d0 0x7faadc1015e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:40.239 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:39.195+0000 7faae0f0c640 1 -- 172.21.15.16:0/1180754280 >> 172.21.15.16:0/1180754280 conn(0x7faadc0fbbc0 msgr2=0x7faadc0fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:52:40.239 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:39.195+0000 7faae0f0c640 1 -- 172.21.15.16:0/1180754280 shutdown_connections 2024-06-18T02:52:40.239 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:39.195+0000 7faae0f0c640 1 -- 172.21.15.16:0/1180754280 wait complete. 2024-06-18T02:52:40.239 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:39.195+0000 7faae0f0c640 1 Processor -- start 2024-06-18T02:52:40.239 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:39.195+0000 7faae0f0c640 1 -- start start 2024-06-18T02:52:40.239 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:39.196+0000 7faae0f0c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faadc1011d0 0x7faadc110ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:40.239 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:39.196+0000 7faae0f0c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faadc111010 con 0x7faadc1011d0 2024-06-18T02:52:40.239 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:39.196+0000 7faada59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faadc1011d0 0x7faadc110ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:40.239 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:39.196+0000 7faada59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faadc1011d0 0x7faadc110ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54184/0 (socket says 172.21.15.16:54184) 2024-06-18T02:52:40.240 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:39.196+0000 7faada59c640 1 -- 172.21.15.16:0/3423282150 learned_addr learned my addr 172.21.15.16:0/3423282150 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:52:40.240 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:39.196+0000 7faada59c640 1 -- 172.21.15.16:0/3423282150 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faacc009450 con 0x7faadc1011d0 2024-06-18T02:52:40.240 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:39.196+0000 7faada59c640 1 --2- 172.21.15.16:0/3423282150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faadc1011d0 0x7faadc110ad0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7faacc009420 tx=0x7faacc008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:40.240 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:39.196+0000 7faacaffd640 1 -- 172.21.15.16:0/3423282150 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faacc03d5b0 con 0x7faadc1011d0 2024-06-18T02:52:40.240 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:39.196+0000 7faacaffd640 1 -- 172.21.15.16:0/3423282150 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7faacc03dba0 con 0x7faadc1011d0 2024-06-18T02:52:40.240 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:39.196+0000 7faae0f0c640 1 -- 172.21.15.16:0/3423282150 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faadc111210 con 0x7faadc1011d0 2024-06-18T02:52:40.240 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:39.196+0000 7faacaffd640 1 -- 172.21.15.16:0/3423282150 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faacc03bb40 con 0x7faadc1011d0 2024-06-18T02:52:40.240 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:39.196+0000 7faae0f0c640 1 -- 172.21.15.16:0/3423282150 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faadc111680 con 0x7faadc1011d0 2024-06-18T02:52:40.240 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:39.197+0000 7faacaffd640 1 -- 172.21.15.16:0/3423282150 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7faacc039070 con 0x7faadc1011d0 2024-06-18T02:52:40.240 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:39.197+0000 7faacaffd640 1 --2- 172.21.15.16:0/3423282150 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7faac4034810 0x7faac4036cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:40.240 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:39.197+0000 7faae0f0c640 1 -- 172.21.15.16:0/3423282150 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faadc04b480 con 0x7faadc1011d0 2024-06-18T02:52:40.240 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:39.197+0000 7faacaffd640 1 -- 172.21.15.16:0/3423282150 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7faacc067690 con 0x7faadc1011d0 2024-06-18T02:52:40.240 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:39.198+0000 7faad9d9b640 1 --2- 172.21.15.16:0/3423282150 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7faac4034810 0x7faac4036cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:40.241 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:39.198+0000 7faad9d9b640 1 --2- 172.21.15.16:0/3423282150 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7faac4034810 0x7faac4036cc0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7faad0006fd0 tx=0x7faad0006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:40.241 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:39.202+0000 7faacaffd640 1 -- 172.21.15.16:0/3423282150 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7faacc037020 con 0x7faadc1011d0 2024-06-18T02:52:40.241 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:39.407+0000 7faae0f0c640 1 -- 172.21.15.16:0/3423282150 --> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7faadc1127d0 con 0x7faac4034810 2024-06-18T02:52:40.241 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:39.412+0000 7faacaffd640 1 -- 172.21.15.16:0/3423282150 <== mgr.14118 v2:172.21.15.16:6800/1807632468 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7faadc1127d0 con 0x7faac4034810 2024-06-18T02:52:40.241 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:39.414+0000 7faae0f0c640 1 -- 172.21.15.16:0/3423282150 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7faac4034810 msgr2=0x7faac4036cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:40.241 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:39.414+0000 7faae0f0c640 1 --2- 172.21.15.16:0/3423282150 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7faac4034810 0x7faac4036cc0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7faad0006fd0 tx=0x7faad0006db0 comp rx=0 tx=0).stop 2024-06-18T02:52:40.241 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:39.414+0000 7faae0f0c640 1 -- 172.21.15.16:0/3423282150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faadc1011d0 msgr2=0x7faadc110ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:40.241 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:39.414+0000 7faae0f0c640 1 --2- 172.21.15.16:0/3423282150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faadc1011d0 0x7faadc110ad0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7faacc009420 tx=0x7faacc008040 comp rx=0 tx=0).stop 2024-06-18T02:52:40.241 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:39.414+0000 7faae0f0c640 1 -- 172.21.15.16:0/3423282150 shutdown_connections 2024-06-18T02:52:40.241 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:39.414+0000 7faae0f0c640 1 --2- 172.21.15.16:0/3423282150 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7faac4034810 0x7faac4036cc0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:40.241 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:39.414+0000 7faae0f0c640 1 --2- 172.21.15.16:0/3423282150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faadc1011d0 0x7faadc110ad0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:40.241 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:39.414+0000 7faae0f0c640 1 -- 172.21.15.16:0/3423282150 >> 172.21.15.16:0/3423282150 conn(0x7faadc0fbbc0 msgr2=0x7faadc0fca90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:52:40.241 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:39.414+0000 7faae0f0c640 1 -- 172.21.15.16:0/3423282150 shutdown_connections 2024-06-18T02:52:40.242 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:39.414+0000 7faae0f0c640 1 -- 172.21.15.16:0/3423282150 wait complete. 2024-06-18T02:52:40.242 INFO:teuthology.orchestra.run.smithi016.stdout:Deploying mgr service with default placement... 2024-06-18T02:52:40.548 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:40 smithi016 ceph-mon[29214]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-06-18T02:52:40.548 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:40 smithi016 ceph-mon[29214]: Saving service mon spec with placement count:5 2024-06-18T02:52:40.548 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:40 smithi016 ceph-mon[29214]: from='mgr.14118 172.21.15.16:0/47051786' entity='mgr.smithi016.amxydz' 2024-06-18T02:52:42.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:41 smithi016 ceph-mon[29214]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-06-18T02:52:42.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:41 smithi016 ceph-mon[29214]: Saving service mgr spec with placement count:2 2024-06-18T02:52:42.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:41 smithi016 ceph-mon[29214]: from='mgr.14118 172.21.15.16:0/47051786' entity='mgr.smithi016.amxydz' 2024-06-18T02:52:42.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:41 smithi016 ceph-mon[29214]: from='mgr.14118 172.21.15.16:0/47051786' entity='mgr.smithi016.amxydz' 2024-06-18T02:52:42.150 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-06-18T02:52:42.150 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:40.626+0000 7f0c6b851640 1 Processor -- start 2024-06-18T02:52:42.150 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:40.626+0000 7f0c6b851640 1 -- start start 2024-06-18T02:52:42.150 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:40.627+0000 7f0c6b851640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c64075ba0 0x7f0c64104940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:42.151 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:40.627+0000 7f0c6b851640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c64104e80 con 0x7f0c64075ba0 2024-06-18T02:52:42.151 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:40.627+0000 7f0c695ed640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c64075ba0 0x7f0c64104940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:42.151 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:40.627+0000 7f0c695ed640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c64075ba0 0x7f0c64104940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54190/0 (socket says 172.21.15.16:54190) 2024-06-18T02:52:42.151 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:40.627+0000 7f0c695ed640 1 -- 172.21.15.16:0/1006539157 learned_addr learned my addr 172.21.15.16:0/1006539157 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:52:42.151 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:40.627+0000 7f0c695ed640 1 -- 172.21.15.16:0/1006539157 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c64104fc0 con 0x7f0c64075ba0 2024-06-18T02:52:42.151 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:40.628+0000 7f0c695ed640 1 --2- 172.21.15.16:0/1006539157 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c64075ba0 0x7f0c64104940 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c009840 tx=0x7f0c5c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7b47da4367aeccd server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:42.151 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:40.628+0000 7f0c63fff640 1 -- 172.21.15.16:0/1006539157 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c5c032440 con 0x7f0c64075ba0 2024-06-18T02:52:42.152 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:40.628+0000 7f0c63fff640 1 -- 172.21.15.16:0/1006539157 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f0c5c032a30 con 0x7f0c64075ba0 2024-06-18T02:52:42.152 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:40.628+0000 7f0c63fff640 1 -- 172.21.15.16:0/1006539157 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c5c031ad0 con 0x7f0c64075ba0 2024-06-18T02:52:42.152 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:40.628+0000 7f0c6b851640 1 -- 172.21.15.16:0/1006539157 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c64075ba0 msgr2=0x7f0c64104940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:42.152 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:40.628+0000 7f0c6b851640 1 --2- 172.21.15.16:0/1006539157 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c64075ba0 0x7f0c64104940 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c009840 tx=0x7f0c5c0091d0 comp rx=0 tx=0).stop 2024-06-18T02:52:42.152 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:40.629+0000 7f0c6b851640 1 -- 172.21.15.16:0/1006539157 shutdown_connections 2024-06-18T02:52:42.152 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:40.629+0000 7f0c6b851640 1 --2- 172.21.15.16:0/1006539157 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c64075ba0 0x7f0c64104940 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:42.152 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:40.629+0000 7f0c6b851640 1 -- 172.21.15.16:0/1006539157 >> 172.21.15.16:0/1006539157 conn(0x7f0c640fc430 msgr2=0x7f0c640fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:52:42.152 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:40.629+0000 7f0c6b851640 1 -- 172.21.15.16:0/1006539157 shutdown_connections 2024-06-18T02:52:42.152 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:40.629+0000 7f0c6b851640 1 -- 172.21.15.16:0/1006539157 wait complete. 2024-06-18T02:52:42.152 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:40.629+0000 7f0c6b851640 1 Processor -- start 2024-06-18T02:52:42.152 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:40.629+0000 7f0c6b851640 1 -- start start 2024-06-18T02:52:42.152 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:40.629+0000 7f0c6b851640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c64075ba0 0x7f0c64110b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:42.152 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:40.629+0000 7f0c6b851640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c64111050 con 0x7f0c64075ba0 2024-06-18T02:52:42.153 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:40.630+0000 7f0c695ed640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c64075ba0 0x7f0c64110b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:42.153 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:40.630+0000 7f0c695ed640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c64075ba0 0x7f0c64110b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54200/0 (socket says 172.21.15.16:54200) 2024-06-18T02:52:42.153 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:40.630+0000 7f0c695ed640 1 -- 172.21.15.16:0/438157415 learned_addr learned my addr 172.21.15.16:0/438157415 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:52:42.153 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:40.630+0000 7f0c695ed640 1 -- 172.21.15.16:0/438157415 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c5c009450 con 0x7f0c64075ba0 2024-06-18T02:52:42.153 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:40.630+0000 7f0c695ed640 1 --2- 172.21.15.16:0/438157415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c64075ba0 0x7f0c64110b10 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c031b90 tx=0x7f0c5c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:42.153 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:40.630+0000 7f0c61ffb640 1 -- 172.21.15.16:0/438157415 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c5c0328b0 con 0x7f0c64075ba0 2024-06-18T02:52:42.153 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:40.630+0000 7f0c6b851640 1 -- 172.21.15.16:0/438157415 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c64111250 con 0x7f0c64075ba0 2024-06-18T02:52:42.153 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:40.630+0000 7f0c61ffb640 1 -- 172.21.15.16:0/438157415 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f0c5c03d8e0 con 0x7f0c64075ba0 2024-06-18T02:52:42.153 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:40.630+0000 7f0c61ffb640 1 -- 172.21.15.16:0/438157415 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c5c03ba20 con 0x7f0c64075ba0 2024-06-18T02:52:42.153 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:40.630+0000 7f0c6b851640 1 -- 172.21.15.16:0/438157415 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c641116c0 con 0x7f0c64075ba0 2024-06-18T02:52:42.153 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:40.631+0000 7f0c61ffb640 1 -- 172.21.15.16:0/438157415 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f0c5c039070 con 0x7f0c64075ba0 2024-06-18T02:52:42.153 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:40.631+0000 7f0c61ffb640 1 --2- 172.21.15.16:0/438157415 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7f0c4c034810 0x7f0c4c036cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:42.153 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:40.631+0000 7f0c61ffb640 1 -- 172.21.15.16:0/438157415 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f0c5c0676a0 con 0x7f0c64075ba0 2024-06-18T02:52:42.154 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:40.632+0000 7f0c68dec640 1 --2- 172.21.15.16:0/438157415 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7f0c4c034810 0x7f0c4c036cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:42.154 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:40.632+0000 7f0c6b851640 1 -- 172.21.15.16:0/438157415 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c6405ead0 con 0x7f0c64075ba0 2024-06-18T02:52:42.154 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:40.632+0000 7f0c68dec640 1 --2- 172.21.15.16:0/438157415 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7f0c4c034810 0x7f0c4c036cc0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0c58006fd0 tx=0x7f0c58006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:42.154 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:40.636+0000 7f0c61ffb640 1 -- 172.21.15.16:0/438157415 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f0c5c041070 con 0x7f0c64075ba0 2024-06-18T02:52:42.154 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:40.823+0000 7f0c6b851640 1 -- 172.21.15.16:0/438157415 --> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f0c640fd7d0 con 0x7f0c4c034810 2024-06-18T02:52:42.154 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:40.827+0000 7f0c61ffb640 1 -- 172.21.15.16:0/438157415 <== mgr.14118 v2:172.21.15.16:6800/1807632468 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f0c640fd7d0 con 0x7f0c4c034810 2024-06-18T02:52:42.154 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:40.829+0000 7f0c6b851640 1 -- 172.21.15.16:0/438157415 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7f0c4c034810 msgr2=0x7f0c4c036cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:42.154 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:40.829+0000 7f0c6b851640 1 --2- 172.21.15.16:0/438157415 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7f0c4c034810 0x7f0c4c036cc0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0c58006fd0 tx=0x7f0c58006db0 comp rx=0 tx=0).stop 2024-06-18T02:52:42.154 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:40.829+0000 7f0c6b851640 1 -- 172.21.15.16:0/438157415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c64075ba0 msgr2=0x7f0c64110b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:42.154 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:40.829+0000 7f0c6b851640 1 --2- 172.21.15.16:0/438157415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c64075ba0 0x7f0c64110b10 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c031b90 tx=0x7f0c5c008040 comp rx=0 tx=0).stop 2024-06-18T02:52:42.154 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:40.829+0000 7f0c6b851640 1 -- 172.21.15.16:0/438157415 shutdown_connections 2024-06-18T02:52:42.154 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:40.829+0000 7f0c6b851640 1 --2- 172.21.15.16:0/438157415 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7f0c4c034810 0x7f0c4c036cc0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:42.154 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:40.829+0000 7f0c6b851640 1 --2- 172.21.15.16:0/438157415 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c64075ba0 0x7f0c64110b10 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:42.155 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:40.829+0000 7f0c6b851640 1 -- 172.21.15.16:0/438157415 >> 172.21.15.16:0/438157415 conn(0x7f0c640fc430 msgr2=0x7f0c640fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:52:42.155 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:40.829+0000 7f0c6b851640 1 -- 172.21.15.16:0/438157415 shutdown_connections 2024-06-18T02:52:42.155 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:40.829+0000 7f0c6b851640 1 -- 172.21.15.16:0/438157415 wait complete. 2024-06-18T02:52:42.155 INFO:teuthology.orchestra.run.smithi016.stdout:Deploying crash service with default placement... 2024-06-18T02:52:43.509 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:43 smithi016 ceph-mon[29214]: from='mgr.14118 172.21.15.16:0/47051786' entity='mgr.smithi016.amxydz' 2024-06-18T02:52:43.509 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:43 smithi016 ceph-mon[29214]: from='mgr.14118 172.21.15.16:0/47051786' entity='mgr.smithi016.amxydz' 2024-06-18T02:52:44.034 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout Scheduled crash update... 2024-06-18T02:52:44.034 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:42.555+0000 7f4554b2f640 1 Processor -- start 2024-06-18T02:52:44.034 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:42.555+0000 7f4554b2f640 1 -- start start 2024-06-18T02:52:44.034 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:42.555+0000 7f4554b2f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4550101d90 0x7f45501021a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:44.034 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:42.555+0000 7f4554b2f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45501026e0 con 0x7f4550101d90 2024-06-18T02:52:44.035 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:42.556+0000 7f454e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4550101d90 0x7f45501021a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:44.035 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:42.556+0000 7f454e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4550101d90 0x7f45501021a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42980/0 (socket says 172.21.15.16:42980) 2024-06-18T02:52:44.035 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:42.556+0000 7f454e59c640 1 -- 172.21.15.16:0/975199857 learned_addr learned my addr 172.21.15.16:0/975199857 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:52:44.035 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:42.556+0000 7f454e59c640 1 -- 172.21.15.16:0/975199857 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4550102820 con 0x7f4550101d90 2024-06-18T02:52:44.035 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:42.556+0000 7f454e59c640 1 --2- 172.21.15.16:0/975199857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4550101d90 0x7f45501021a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4540009840 tx=0x7f45400091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b5abe6d57ae7875 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:44.035 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:42.557+0000 7f454d59a640 1 -- 172.21.15.16:0/975199857 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4540032440 con 0x7f4550101d90 2024-06-18T02:52:44.035 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:42.557+0000 7f454d59a640 1 -- 172.21.15.16:0/975199857 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4540032a70 con 0x7f4550101d90 2024-06-18T02:52:44.035 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:42.557+0000 7f454d59a640 1 -- 172.21.15.16:0/975199857 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4540031ad0 con 0x7f4550101d90 2024-06-18T02:52:44.035 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:42.557+0000 7f4554b2f640 1 -- 172.21.15.16:0/975199857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4550101d90 msgr2=0x7f45501021a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:44.035 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:42.557+0000 7f4554b2f640 1 --2- 172.21.15.16:0/975199857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4550101d90 0x7f45501021a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4540009840 tx=0x7f45400091d0 comp rx=0 tx=0).stop 2024-06-18T02:52:44.035 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:42.557+0000 7f4554b2f640 1 -- 172.21.15.16:0/975199857 shutdown_connections 2024-06-18T02:52:44.035 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:42.557+0000 7f4554b2f640 1 --2- 172.21.15.16:0/975199857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4550101d90 0x7f45501021a0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:44.035 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:42.557+0000 7f4554b2f640 1 -- 172.21.15.16:0/975199857 >> 172.21.15.16:0/975199857 conn(0x7f45500fd590 msgr2=0x7f45500ff9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:52:44.036 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:42.557+0000 7f4554b2f640 1 -- 172.21.15.16:0/975199857 shutdown_connections 2024-06-18T02:52:44.036 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:42.557+0000 7f4554b2f640 1 -- 172.21.15.16:0/975199857 wait complete. 2024-06-18T02:52:44.036 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:42.558+0000 7f4554b2f640 1 Processor -- start 2024-06-18T02:52:44.036 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:42.558+0000 7f4554b2f640 1 -- start start 2024-06-18T02:52:44.036 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:42.558+0000 7f4554b2f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4550101d90 0x7f4550110f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:44.036 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:42.558+0000 7f4554b2f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4550111440 con 0x7f4550101d90 2024-06-18T02:52:44.036 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:42.558+0000 7f454e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4550101d90 0x7f4550110f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:44.036 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:42.558+0000 7f454e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4550101d90 0x7f4550110f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42990/0 (socket says 172.21.15.16:42990) 2024-06-18T02:52:44.036 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:42.558+0000 7f454e59c640 1 -- 172.21.15.16:0/1378308022 learned_addr learned my addr 172.21.15.16:0/1378308022 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:52:44.036 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:42.558+0000 7f454e59c640 1 -- 172.21.15.16:0/1378308022 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4540009450 con 0x7f4550101d90 2024-06-18T02:52:44.036 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:42.559+0000 7f454e59c640 1 --2- 172.21.15.16:0/1378308022 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4550101d90 0x7f4550110f00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4540009810 tx=0x7f454002c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:44.036 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:42.559+0000 7f453effd640 1 -- 172.21.15.16:0/1378308022 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f45400328b0 con 0x7f4550101d90 2024-06-18T02:52:44.036 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:42.559+0000 7f453effd640 1 -- 172.21.15.16:0/1378308022 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f454003b910 con 0x7f4550101d90 2024-06-18T02:52:44.036 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:42.559+0000 7f453effd640 1 -- 172.21.15.16:0/1378308022 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f454003a980 con 0x7f4550101d90 2024-06-18T02:52:44.037 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:42.559+0000 7f4554b2f640 1 -- 172.21.15.16:0/1378308022 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4550111640 con 0x7f4550101d90 2024-06-18T02:52:44.037 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:42.559+0000 7f4554b2f640 1 -- 172.21.15.16:0/1378308022 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45501120f0 con 0x7f4550101d90 2024-06-18T02:52:44.037 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:42.560+0000 7f453effd640 1 -- 172.21.15.16:0/1378308022 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f4540039070 con 0x7f4550101d90 2024-06-18T02:52:44.037 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:42.560+0000 7f453effd640 1 --2- 172.21.15.16:0/1378308022 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7f4538034860 0x7f4538036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:44.037 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:42.560+0000 7f4554b2f640 1 -- 172.21.15.16:0/1378308022 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f455005ead0 con 0x7f4550101d90 2024-06-18T02:52:44.037 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:42.560+0000 7f453effd640 1 -- 172.21.15.16:0/1378308022 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f4540066dc0 con 0x7f4550101d90 2024-06-18T02:52:44.037 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:42.560+0000 7f454dd9b640 1 --2- 172.21.15.16:0/1378308022 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7f4538034860 0x7f4538036d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:44.037 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:42.560+0000 7f454dd9b640 1 --2- 172.21.15.16:0/1378308022 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7f4538034860 0x7f4538036d10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4544006fd0 tx=0x7f4544006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:44.037 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:42.564+0000 7f453effd640 1 -- 172.21.15.16:0/1378308022 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f454003f080 con 0x7f4550101d90 2024-06-18T02:52:44.037 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:42.762+0000 7f4554b2f640 1 -- 172.21.15.16:0/1378308022 --> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f4550103ce0 con 0x7f4538034860 2024-06-18T02:52:44.037 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:42.766+0000 7f453effd640 1 -- 172.21.15.16:0/1378308022 <== mgr.14118 v2:172.21.15.16:6800/1807632468 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f4550103ce0 con 0x7f4538034860 2024-06-18T02:52:44.037 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:42.768+0000 7f4554b2f640 1 -- 172.21.15.16:0/1378308022 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7f4538034860 msgr2=0x7f4538036d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:44.037 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:42.768+0000 7f4554b2f640 1 --2- 172.21.15.16:0/1378308022 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7f4538034860 0x7f4538036d10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4544006fd0 tx=0x7f4544006db0 comp rx=0 tx=0).stop 2024-06-18T02:52:44.038 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:42.768+0000 7f4554b2f640 1 -- 172.21.15.16:0/1378308022 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4550101d90 msgr2=0x7f4550110f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:44.038 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:42.768+0000 7f4554b2f640 1 --2- 172.21.15.16:0/1378308022 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4550101d90 0x7f4550110f00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4540009810 tx=0x7f454002c040 comp rx=0 tx=0).stop 2024-06-18T02:52:44.038 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:42.768+0000 7f4554b2f640 1 -- 172.21.15.16:0/1378308022 shutdown_connections 2024-06-18T02:52:44.038 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:42.768+0000 7f4554b2f640 1 --2- 172.21.15.16:0/1378308022 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7f4538034860 0x7f4538036d10 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:44.038 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:42.768+0000 7f4554b2f640 1 --2- 172.21.15.16:0/1378308022 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4550101d90 0x7f4550110f00 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:44.038 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:42.768+0000 7f4554b2f640 1 -- 172.21.15.16:0/1378308022 >> 172.21.15.16:0/1378308022 conn(0x7f45500fd590 msgr2=0x7f45500fe3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:52:44.038 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:42.768+0000 7f4554b2f640 1 -- 172.21.15.16:0/1378308022 shutdown_connections 2024-06-18T02:52:44.038 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:42.768+0000 7f4554b2f640 1 -- 172.21.15.16:0/1378308022 wait complete. 2024-06-18T02:52:44.038 INFO:teuthology.orchestra.run.smithi016.stdout:Deploying prometheus service with default placement... 2024-06-18T02:52:44.631 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:44 smithi016 ceph-mon[29214]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-06-18T02:52:44.631 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:44 smithi016 ceph-mon[29214]: Saving service crash spec with placement * 2024-06-18T02:52:44.631 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:44 smithi016 ceph-mon[29214]: from='mgr.14118 172.21.15.16:0/47051786' entity='mgr.smithi016.amxydz' 2024-06-18T02:52:45.224 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout Scheduled prometheus update... 2024-06-18T02:52:45.224 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:44.426+0000 7ff71d123640 1 Processor -- start 2024-06-18T02:52:45.224 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:44.426+0000 7ff71d123640 1 -- start start 2024-06-18T02:52:45.224 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:44.427+0000 7ff71d123640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff718101d50 0x7ff718102160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:45.224 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:44.427+0000 7ff71d123640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7181026a0 con 0x7ff718101d50 2024-06-18T02:52:45.225 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:44.427+0000 7ff716d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff718101d50 0x7ff718102160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:45.225 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:44.427+0000 7ff716d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff718101d50 0x7ff718102160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42994/0 (socket says 172.21.15.16:42994) 2024-06-18T02:52:45.225 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:44.427+0000 7ff716d9d640 1 -- 172.21.15.16:0/1318851655 learned_addr learned my addr 172.21.15.16:0/1318851655 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:52:45.225 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:44.427+0000 7ff716d9d640 1 -- 172.21.15.16:0/1318851655 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7181027e0 con 0x7ff718101d50 2024-06-18T02:52:45.225 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:44.428+0000 7ff716d9d640 1 --2- 172.21.15.16:0/1318851655 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff718101d50 0x7ff718102160 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff708009840 tx=0x7ff7080091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66e9d237662df03 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:45.225 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:44.428+0000 7ff71659c640 1 -- 172.21.15.16:0/1318851655 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff708032440 con 0x7ff718101d50 2024-06-18T02:52:45.225 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:44.428+0000 7ff71659c640 1 -- 172.21.15.16:0/1318851655 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff708032a70 con 0x7ff718101d50 2024-06-18T02:52:45.225 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:44.428+0000 7ff71659c640 1 -- 172.21.15.16:0/1318851655 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff708031ad0 con 0x7ff718101d50 2024-06-18T02:52:45.225 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:44.429+0000 7ff71d123640 1 -- 172.21.15.16:0/1318851655 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff718101d50 msgr2=0x7ff718102160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:45.225 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:44.429+0000 7ff71d123640 1 --2- 172.21.15.16:0/1318851655 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff718101d50 0x7ff718102160 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff708009840 tx=0x7ff7080091d0 comp rx=0 tx=0).stop 2024-06-18T02:52:45.225 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:44.429+0000 7ff71d123640 1 -- 172.21.15.16:0/1318851655 shutdown_connections 2024-06-18T02:52:45.225 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:44.429+0000 7ff71d123640 1 --2- 172.21.15.16:0/1318851655 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff718101d50 0x7ff718102160 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:45.226 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:44.429+0000 7ff71d123640 1 -- 172.21.15.16:0/1318851655 >> 172.21.15.16:0/1318851655 conn(0x7ff7180fd550 msgr2=0x7ff7180ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:52:45.226 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:44.429+0000 7ff71d123640 1 -- 172.21.15.16:0/1318851655 shutdown_connections 2024-06-18T02:52:45.226 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:44.429+0000 7ff71d123640 1 -- 172.21.15.16:0/1318851655 wait complete. 2024-06-18T02:52:45.226 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:44.429+0000 7ff71d123640 1 Processor -- start 2024-06-18T02:52:45.226 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:44.429+0000 7ff71d123640 1 -- start start 2024-06-18T02:52:45.226 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:44.430+0000 7ff71d123640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff718101d50 0x7ff718077c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:45.226 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:44.430+0000 7ff71d123640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff718078150 con 0x7ff718101d50 2024-06-18T02:52:45.226 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:44.430+0000 7ff716d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff718101d50 0x7ff718077c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:45.226 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:44.430+0000 7ff716d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff718101d50 0x7ff718077c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43010/0 (socket says 172.21.15.16:43010) 2024-06-18T02:52:45.226 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:44.430+0000 7ff716d9d640 1 -- 172.21.15.16:0/2416463058 learned_addr learned my addr 172.21.15.16:0/2416463058 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:52:45.226 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:44.430+0000 7ff716d9d640 1 -- 172.21.15.16:0/2416463058 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff708009450 con 0x7ff718101d50 2024-06-18T02:52:45.226 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:44.430+0000 7ff716d9d640 1 --2- 172.21.15.16:0/2416463058 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff718101d50 0x7ff718077c10 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff708009810 tx=0x7ff70802c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:45.226 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:44.430+0000 7ff70f7fe640 1 -- 172.21.15.16:0/2416463058 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff7080328b0 con 0x7ff718101d50 2024-06-18T02:52:45.227 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:44.430+0000 7ff70f7fe640 1 -- 172.21.15.16:0/2416463058 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff70803b910 con 0x7ff718101d50 2024-06-18T02:52:45.227 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:44.430+0000 7ff71d123640 1 -- 172.21.15.16:0/2416463058 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7180761e0 con 0x7ff718101d50 2024-06-18T02:52:45.227 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:44.430+0000 7ff70f7fe640 1 -- 172.21.15.16:0/2416463058 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff70803a980 con 0x7ff718101d50 2024-06-18T02:52:45.227 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:44.430+0000 7ff71d123640 1 -- 172.21.15.16:0/2416463058 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff718076650 con 0x7ff718101d50 2024-06-18T02:52:45.227 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:44.431+0000 7ff70f7fe640 1 -- 172.21.15.16:0/2416463058 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7ff708039070 con 0x7ff718101d50 2024-06-18T02:52:45.227 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:44.431+0000 7ff71d123640 1 -- 172.21.15.16:0/2416463058 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff71804b480 con 0x7ff718101d50 2024-06-18T02:52:45.227 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:44.431+0000 7ff70f7fe640 1 --2- 172.21.15.16:0/2416463058 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7ff704034810 0x7ff704036cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:45.227 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:44.432+0000 7ff70f7fe640 1 -- 172.21.15.16:0/2416463058 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7ff708066df0 con 0x7ff718101d50 2024-06-18T02:52:45.227 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:44.432+0000 7ff70ffff640 1 --2- 172.21.15.16:0/2416463058 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7ff704034810 0x7ff704036cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:45.227 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:44.432+0000 7ff70ffff640 1 --2- 172.21.15.16:0/2416463058 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7ff704034810 0x7ff704036cc0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff700006fd0 tx=0x7ff700006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:45.227 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:44.436+0000 7ff70f7fe640 1 -- 172.21.15.16:0/2416463058 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7ff708039320 con 0x7ff718101d50 2024-06-18T02:52:45.227 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:44.631+0000 7ff71d123640 1 -- 172.21.15.16:0/2416463058 --> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7ff718076e60 con 0x7ff704034810 2024-06-18T02:52:45.227 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:44.636+0000 7ff70f7fe640 1 -- 172.21.15.16:0/2416463058 <== mgr.14118 v2:172.21.15.16:6800/1807632468 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7ff718076e60 con 0x7ff704034810 2024-06-18T02:52:45.228 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:44.638+0000 7ff71d123640 1 -- 172.21.15.16:0/2416463058 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7ff704034810 msgr2=0x7ff704036cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:45.228 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:44.638+0000 7ff71d123640 1 --2- 172.21.15.16:0/2416463058 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7ff704034810 0x7ff704036cc0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff700006fd0 tx=0x7ff700006db0 comp rx=0 tx=0).stop 2024-06-18T02:52:45.228 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:44.638+0000 7ff71d123640 1 -- 172.21.15.16:0/2416463058 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff718101d50 msgr2=0x7ff718077c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:45.228 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:44.638+0000 7ff71d123640 1 --2- 172.21.15.16:0/2416463058 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff718101d50 0x7ff718077c10 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff708009810 tx=0x7ff70802c040 comp rx=0 tx=0).stop 2024-06-18T02:52:45.228 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:44.638+0000 7ff71d123640 1 -- 172.21.15.16:0/2416463058 shutdown_connections 2024-06-18T02:52:45.228 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:44.638+0000 7ff71d123640 1 --2- 172.21.15.16:0/2416463058 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7ff704034810 0x7ff704036cc0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:45.228 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:44.638+0000 7ff71d123640 1 --2- 172.21.15.16:0/2416463058 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff718101d50 0x7ff718077c10 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:45.228 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:44.638+0000 7ff71d123640 1 -- 172.21.15.16:0/2416463058 >> 172.21.15.16:0/2416463058 conn(0x7ff7180fd550 msgr2=0x7ff7180fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:52:45.228 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:44.638+0000 7ff71d123640 1 -- 172.21.15.16:0/2416463058 shutdown_connections 2024-06-18T02:52:45.228 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:44.638+0000 7ff71d123640 1 -- 172.21.15.16:0/2416463058 wait complete. 2024-06-18T02:52:45.228 INFO:teuthology.orchestra.run.smithi016.stdout:Deploying grafana service with default placement... 2024-06-18T02:52:45.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:45 smithi016 ceph-mon[29214]: from='mgr.14118 172.21.15.16:0/47051786' entity='mgr.smithi016.amxydz' 2024-06-18T02:52:45.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:45 smithi016 ceph-mon[29214]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-06-18T02:52:45.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:45 smithi016 ceph-mon[29214]: Saving service prometheus spec with placement count:1 2024-06-18T02:52:45.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:45 smithi016 ceph-mon[29214]: from='mgr.14118 172.21.15.16:0/47051786' entity='mgr.smithi016.amxydz' 2024-06-18T02:52:47.171 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout Scheduled grafana update... 2024-06-18T02:52:47.171 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:46.165+0000 7fb4a70e1640 1 Processor -- start 2024-06-18T02:52:47.171 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:46.165+0000 7fb4a70e1640 1 -- start start 2024-06-18T02:52:47.171 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:46.165+0000 7fb4a70e1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4a0101cf0 0x7fb4a0102100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:47.171 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:46.165+0000 7fb4a70e1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a0102640 con 0x7fb4a0101cf0 2024-06-18T02:52:47.171 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:46.165+0000 7fb4a4e7d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4a0101cf0 0x7fb4a0102100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:47.171 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:46.165+0000 7fb4a4e7d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4a0101cf0 0x7fb4a0102100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43022/0 (socket says 172.21.15.16:43022) 2024-06-18T02:52:47.171 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:46.165+0000 7fb4a4e7d640 1 -- 172.21.15.16:0/453365630 learned_addr learned my addr 172.21.15.16:0/453365630 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:52:47.171 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:46.166+0000 7fb4a4e7d640 1 -- 172.21.15.16:0/453365630 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4a0102780 con 0x7fb4a0101cf0 2024-06-18T02:52:47.171 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:46.166+0000 7fb4a4e7d640 1 --2- 172.21.15.16:0/453365630 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4a0101cf0 0x7fb4a0102100 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb494009840 tx=0x7fb4940091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66131e8eca2b71a3 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:47.172 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:46.166+0000 7fb49f7fe640 1 -- 172.21.15.16:0/453365630 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb494032440 con 0x7fb4a0101cf0 2024-06-18T02:52:47.172 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:46.167+0000 7fb49f7fe640 1 -- 172.21.15.16:0/453365630 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb494032a70 con 0x7fb4a0101cf0 2024-06-18T02:52:47.172 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:46.167+0000 7fb49f7fe640 1 -- 172.21.15.16:0/453365630 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb494031ad0 con 0x7fb4a0101cf0 2024-06-18T02:52:47.172 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:46.167+0000 7fb4a70e1640 1 -- 172.21.15.16:0/453365630 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4a0101cf0 msgr2=0x7fb4a0102100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:47.172 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:46.167+0000 7fb4a70e1640 1 --2- 172.21.15.16:0/453365630 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4a0101cf0 0x7fb4a0102100 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb494009840 tx=0x7fb4940091d0 comp rx=0 tx=0).stop 2024-06-18T02:52:47.172 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:46.167+0000 7fb4a70e1640 1 -- 172.21.15.16:0/453365630 shutdown_connections 2024-06-18T02:52:47.172 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:46.167+0000 7fb4a70e1640 1 --2- 172.21.15.16:0/453365630 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4a0101cf0 0x7fb4a0102100 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:47.172 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:46.167+0000 7fb4a70e1640 1 -- 172.21.15.16:0/453365630 >> 172.21.15.16:0/453365630 conn(0x7fb4a00fd4d0 msgr2=0x7fb4a00ff8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:52:47.172 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:46.167+0000 7fb4a70e1640 1 -- 172.21.15.16:0/453365630 shutdown_connections 2024-06-18T02:52:47.172 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:46.167+0000 7fb4a70e1640 1 -- 172.21.15.16:0/453365630 wait complete. 2024-06-18T02:52:47.172 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:46.168+0000 7fb4a70e1640 1 Processor -- start 2024-06-18T02:52:47.172 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:46.168+0000 7fb4a70e1640 1 -- start start 2024-06-18T02:52:47.172 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:46.168+0000 7fb4a70e1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4a0101cf0 0x7fb4a0110b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:47.172 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:46.168+0000 7fb4a70e1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a01110d0 con 0x7fb4a0101cf0 2024-06-18T02:52:47.173 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:46.168+0000 7fb4a4e7d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4a0101cf0 0x7fb4a0110b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:47.173 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:46.168+0000 7fb4a4e7d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4a0101cf0 0x7fb4a0110b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43034/0 (socket says 172.21.15.16:43034) 2024-06-18T02:52:47.173 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:46.168+0000 7fb4a4e7d640 1 -- 172.21.15.16:0/751253586 learned_addr learned my addr 172.21.15.16:0/751253586 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:52:47.173 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:46.169+0000 7fb4a4e7d640 1 -- 172.21.15.16:0/751253586 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb494009450 con 0x7fb4a0101cf0 2024-06-18T02:52:47.173 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:46.169+0000 7fb4a4e7d640 1 --2- 172.21.15.16:0/751253586 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4a0101cf0 0x7fb4a0110b90 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb494009810 tx=0x7fb49402c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:47.173 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:46.169+0000 7fb49d7fa640 1 -- 172.21.15.16:0/751253586 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb4940328b0 con 0x7fb4a0101cf0 2024-06-18T02:52:47.173 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:46.169+0000 7fb49d7fa640 1 -- 172.21.15.16:0/751253586 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb49403b910 con 0x7fb4a0101cf0 2024-06-18T02:52:47.173 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:46.169+0000 7fb49d7fa640 1 -- 172.21.15.16:0/751253586 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb49403a980 con 0x7fb4a0101cf0 2024-06-18T02:52:47.174 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:46.169+0000 7fb4a70e1640 1 -- 172.21.15.16:0/751253586 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4a01112d0 con 0x7fb4a0101cf0 2024-06-18T02:52:47.174 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:46.169+0000 7fb4a70e1640 1 -- 172.21.15.16:0/751253586 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4a0111740 con 0x7fb4a0101cf0 2024-06-18T02:52:47.174 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:46.170+0000 7fb49d7fa640 1 -- 172.21.15.16:0/751253586 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7fb494039070 con 0x7fb4a0101cf0 2024-06-18T02:52:47.174 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:46.170+0000 7fb4a70e1640 1 -- 172.21.15.16:0/751253586 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4a004b480 con 0x7fb4a0101cf0 2024-06-18T02:52:47.174 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:46.170+0000 7fb49d7fa640 1 --2- 172.21.15.16:0/751253586 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7fb490034860 0x7fb490036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:47.174 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:46.170+0000 7fb49d7fa640 1 -- 172.21.15.16:0/751253586 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fb494066d70 con 0x7fb4a0101cf0 2024-06-18T02:52:47.174 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:46.170+0000 7fb49ffff640 1 --2- 172.21.15.16:0/751253586 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7fb490034860 0x7fb490036d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:47.174 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:46.171+0000 7fb49ffff640 1 --2- 172.21.15.16:0/751253586 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7fb490034860 0x7fb490036d10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb48c006fd0 tx=0x7fb48c006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:47.174 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:46.175+0000 7fb49d7fa640 1 -- 172.21.15.16:0/751253586 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fb49406b070 con 0x7fb4a0101cf0 2024-06-18T02:52:47.174 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:46.358+0000 7fb4a70e1640 1 -- 172.21.15.16:0/751253586 --> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7fb4a0111740 con 0x7fb490034860 2024-06-18T02:52:47.174 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:46.362+0000 7fb49d7fa640 1 -- 172.21.15.16:0/751253586 <== mgr.14118 v2:172.21.15.16:6800/1807632468 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7fb4a0111740 con 0x7fb490034860 2024-06-18T02:52:47.174 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:46.364+0000 7fb4a70e1640 1 -- 172.21.15.16:0/751253586 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7fb490034860 msgr2=0x7fb490036d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:47.175 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:46.364+0000 7fb4a70e1640 1 --2- 172.21.15.16:0/751253586 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7fb490034860 0x7fb490036d10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb48c006fd0 tx=0x7fb48c006db0 comp rx=0 tx=0).stop 2024-06-18T02:52:47.175 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:46.364+0000 7fb4a70e1640 1 -- 172.21.15.16:0/751253586 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4a0101cf0 msgr2=0x7fb4a0110b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:47.175 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:46.364+0000 7fb4a70e1640 1 --2- 172.21.15.16:0/751253586 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4a0101cf0 0x7fb4a0110b90 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb494009810 tx=0x7fb49402c040 comp rx=0 tx=0).stop 2024-06-18T02:52:47.175 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:46.364+0000 7fb4a70e1640 1 -- 172.21.15.16:0/751253586 shutdown_connections 2024-06-18T02:52:47.175 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:46.364+0000 7fb4a70e1640 1 --2- 172.21.15.16:0/751253586 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7fb490034860 0x7fb490036d10 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:47.175 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:46.364+0000 7fb4a70e1640 1 --2- 172.21.15.16:0/751253586 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4a0101cf0 0x7fb4a0110b90 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:47.175 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:46.364+0000 7fb4a70e1640 1 -- 172.21.15.16:0/751253586 >> 172.21.15.16:0/751253586 conn(0x7fb4a00fd4d0 msgr2=0x7fb4a00fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:52:47.175 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:46.364+0000 7fb4a70e1640 1 -- 172.21.15.16:0/751253586 shutdown_connections 2024-06-18T02:52:47.175 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:46.364+0000 7fb4a70e1640 1 -- 172.21.15.16:0/751253586 wait complete. 2024-06-18T02:52:47.175 INFO:teuthology.orchestra.run.smithi016.stdout:Deploying node-exporter service with default placement... 2024-06-18T02:52:47.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:47 smithi016 ceph-mon[29214]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-06-18T02:52:47.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:47 smithi016 ceph-mon[29214]: Saving service grafana spec with placement count:1 2024-06-18T02:52:47.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:47 smithi016 ceph-mon[29214]: from='mgr.14118 172.21.15.16:0/47051786' entity='mgr.smithi016.amxydz' 2024-06-18T02:52:48.428 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:48 smithi016 ceph-mon[29214]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:52:48.429 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:48 smithi016 ceph-mon[29214]: from='mgr.14118 172.21.15.16:0/47051786' entity='mgr.smithi016.amxydz' 2024-06-18T02:52:48.441 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout Scheduled node-exporter update... 2024-06-18T02:52:48.441 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:47.604+0000 7fe8037a4640 1 Processor -- start 2024-06-18T02:52:48.442 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:47.604+0000 7fe8037a4640 1 -- start start 2024-06-18T02:52:48.442 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:47.604+0000 7fe8037a4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7fc075b40 0x7fe7fc106da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:48.442 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:47.604+0000 7fe8037a4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7fc075f50 con 0x7fe7fc075b40 2024-06-18T02:52:48.442 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:47.605+0000 7fe801540640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7fc075b40 0x7fe7fc106da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:48.442 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:47.605+0000 7fe801540640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7fc075b40 0x7fe7fc106da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43040/0 (socket says 172.21.15.16:43040) 2024-06-18T02:52:48.442 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:47.605+0000 7fe801540640 1 -- 172.21.15.16:0/2378806966 learned_addr learned my addr 172.21.15.16:0/2378806966 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:52:48.443 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:47.605+0000 7fe801540640 1 -- 172.21.15.16:0/2378806966 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7fc102790 con 0x7fe7fc075b40 2024-06-18T02:52:48.443 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:47.606+0000 7fe801540640 1 --2- 172.21.15.16:0/2378806966 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7fc075b40 0x7fe7fc106da0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe7f4009840 tx=0x7fe7f40091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55f6f649396c2c86 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:48.443 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:47.606+0000 7fe7fbfff640 1 -- 172.21.15.16:0/2378806966 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7f4032440 con 0x7fe7fc075b40 2024-06-18T02:52:48.443 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:47.606+0000 7fe7fbfff640 1 -- 172.21.15.16:0/2378806966 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe7f4032a70 con 0x7fe7fc075b40 2024-06-18T02:52:48.443 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:47.606+0000 7fe7fbfff640 1 -- 172.21.15.16:0/2378806966 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7f4031ad0 con 0x7fe7fc075b40 2024-06-18T02:52:48.444 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:47.606+0000 7fe8037a4640 1 -- 172.21.15.16:0/2378806966 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7fc075b40 msgr2=0x7fe7fc106da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:48.444 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:47.606+0000 7fe8037a4640 1 --2- 172.21.15.16:0/2378806966 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7fc075b40 0x7fe7fc106da0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe7f4009840 tx=0x7fe7f40091d0 comp rx=0 tx=0).stop 2024-06-18T02:52:48.444 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:47.606+0000 7fe8037a4640 1 -- 172.21.15.16:0/2378806966 shutdown_connections 2024-06-18T02:52:48.444 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:47.606+0000 7fe8037a4640 1 --2- 172.21.15.16:0/2378806966 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7fc075b40 0x7fe7fc106da0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:48.444 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:47.606+0000 7fe8037a4640 1 -- 172.21.15.16:0/2378806966 >> 172.21.15.16:0/2378806966 conn(0x7fe7fc0fc430 msgr2=0x7fe7fc0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:52:48.444 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:47.607+0000 7fe8037a4640 1 -- 172.21.15.16:0/2378806966 shutdown_connections 2024-06-18T02:52:48.445 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:47.607+0000 7fe8037a4640 1 -- 172.21.15.16:0/2378806966 wait complete. 2024-06-18T02:52:48.445 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:47.607+0000 7fe8037a4640 1 Processor -- start 2024-06-18T02:52:48.445 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:47.607+0000 7fe8037a4640 1 -- start start 2024-06-18T02:52:48.445 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:47.607+0000 7fe8037a4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7fc075b40 0x7fe7fc112e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:48.445 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:47.607+0000 7fe8037a4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7fc113350 con 0x7fe7fc075b40 2024-06-18T02:52:48.445 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:47.607+0000 7fe801540640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7fc075b40 0x7fe7fc112e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:48.446 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:47.607+0000 7fe801540640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7fc075b40 0x7fe7fc112e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43054/0 (socket says 172.21.15.16:43054) 2024-06-18T02:52:48.446 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:47.608+0000 7fe801540640 1 -- 172.21.15.16:0/3018814279 learned_addr learned my addr 172.21.15.16:0/3018814279 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:52:48.446 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:47.608+0000 7fe801540640 1 -- 172.21.15.16:0/3018814279 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7f4009450 con 0x7fe7fc075b40 2024-06-18T02:52:48.446 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:47.608+0000 7fe801540640 1 --2- 172.21.15.16:0/3018814279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7fc075b40 0x7fe7fc112e10 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe7f40098a0 tx=0x7fe7f402c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:48.446 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:47.608+0000 7fe7f9ffb640 1 -- 172.21.15.16:0/3018814279 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7f4031e90 con 0x7fe7fc075b40 2024-06-18T02:52:48.446 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:47.608+0000 7fe7f9ffb640 1 -- 172.21.15.16:0/3018814279 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe7f403b910 con 0x7fe7fc075b40 2024-06-18T02:52:48.447 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:47.608+0000 7fe8037a4640 1 -- 172.21.15.16:0/3018814279 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7fc1135b0 con 0x7fe7fc075b40 2024-06-18T02:52:48.447 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:47.608+0000 7fe7f9ffb640 1 -- 172.21.15.16:0/3018814279 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7f403a9b0 con 0x7fe7fc075b40 2024-06-18T02:52:48.447 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:47.608+0000 7fe8037a4640 1 -- 172.21.15.16:0/3018814279 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7fc113fa0 con 0x7fe7fc075b40 2024-06-18T02:52:48.447 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:47.609+0000 7fe7f9ffb640 1 -- 172.21.15.16:0/3018814279 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7fe7f4039070 con 0x7fe7fc075b40 2024-06-18T02:52:48.447 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:47.609+0000 7fe8037a4640 1 -- 172.21.15.16:0/3018814279 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7fc05ead0 con 0x7fe7fc075b40 2024-06-18T02:52:48.447 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:47.609+0000 7fe7f9ffb640 1 --2- 172.21.15.16:0/3018814279 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7fe7e4034860 0x7fe7e4036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:48.448 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:47.609+0000 7fe7f9ffb640 1 -- 172.21.15.16:0/3018814279 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fe7f40670e0 con 0x7fe7fc075b40 2024-06-18T02:52:48.448 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:47.610+0000 7fe800d3f640 1 --2- 172.21.15.16:0/3018814279 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7fe7e4034860 0x7fe7e4036d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:48.448 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:47.610+0000 7fe800d3f640 1 --2- 172.21.15.16:0/3018814279 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7fe7e4034860 0x7fe7e4036d10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fe7f0006fd0 tx=0x7fe7f0006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:48.448 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:47.614+0000 7fe7f9ffb640 1 -- 172.21.15.16:0/3018814279 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fe7f4030930 con 0x7fe7fc075b40 2024-06-18T02:52:48.448 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:47.817+0000 7fe8037a4640 1 -- 172.21.15.16:0/3018814279 --> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7fe7fc0fd850 con 0x7fe7e4034860 2024-06-18T02:52:48.448 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:47.821+0000 7fe7f9ffb640 1 -- 172.21.15.16:0/3018814279 <== mgr.14118 v2:172.21.15.16:6800/1807632468 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7fe7fc0fd850 con 0x7fe7e4034860 2024-06-18T02:52:48.448 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:47.823+0000 7fe8037a4640 1 -- 172.21.15.16:0/3018814279 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7fe7e4034860 msgr2=0x7fe7e4036d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:48.449 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:47.823+0000 7fe8037a4640 1 --2- 172.21.15.16:0/3018814279 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7fe7e4034860 0x7fe7e4036d10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fe7f0006fd0 tx=0x7fe7f0006db0 comp rx=0 tx=0).stop 2024-06-18T02:52:48.449 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:47.823+0000 7fe8037a4640 1 -- 172.21.15.16:0/3018814279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7fc075b40 msgr2=0x7fe7fc112e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:48.449 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:47.823+0000 7fe8037a4640 1 --2- 172.21.15.16:0/3018814279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7fc075b40 0x7fe7fc112e10 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe7f40098a0 tx=0x7fe7f402c040 comp rx=0 tx=0).stop 2024-06-18T02:52:48.449 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:47.823+0000 7fe8037a4640 1 -- 172.21.15.16:0/3018814279 shutdown_connections 2024-06-18T02:52:48.449 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:47.823+0000 7fe8037a4640 1 --2- 172.21.15.16:0/3018814279 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7fe7e4034860 0x7fe7e4036d10 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:48.449 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:47.823+0000 7fe8037a4640 1 --2- 172.21.15.16:0/3018814279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7fc075b40 0x7fe7fc112e10 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:48.449 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:47.823+0000 7fe8037a4640 1 -- 172.21.15.16:0/3018814279 >> 172.21.15.16:0/3018814279 conn(0x7fe7fc0fc430 msgr2=0x7fe7fc0fd060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:52:48.449 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:47.823+0000 7fe8037a4640 1 -- 172.21.15.16:0/3018814279 shutdown_connections 2024-06-18T02:52:48.450 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:47.823+0000 7fe8037a4640 1 -- 172.21.15.16:0/3018814279 wait complete. 2024-06-18T02:52:48.450 INFO:teuthology.orchestra.run.smithi016.stdout:Deploying alertmanager service with default placement... 2024-06-18T02:52:49.609 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:49 smithi016 ceph-mon[29214]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-06-18T02:52:49.610 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:49 smithi016 ceph-mon[29214]: Saving service node-exporter spec with placement * 2024-06-18T02:52:49.610 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:49 smithi016 ceph-mon[29214]: from='mgr.14118 172.21.15.16:0/47051786' entity='mgr.smithi016.amxydz' 2024-06-18T02:52:49.618 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout Scheduled alertmanager update... 2024-06-18T02:52:49.618 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:48.852+0000 7f6e9623c640 1 Processor -- start 2024-06-18T02:52:49.619 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:48.852+0000 7f6e9623c640 1 -- start start 2024-06-18T02:52:49.619 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:48.852+0000 7f6e9623c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e90101d90 0x7f6e901021a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:49.619 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:48.852+0000 7f6e9623c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e901026e0 con 0x7f6e90101d90 2024-06-18T02:52:49.619 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:48.852+0000 7f6e8f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e90101d90 0x7f6e901021a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:49.619 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:48.852+0000 7f6e8f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e90101d90 0x7f6e901021a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43058/0 (socket says 172.21.15.16:43058) 2024-06-18T02:52:49.619 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:48.852+0000 7f6e8f7fe640 1 -- 172.21.15.16:0/4153633360 learned_addr learned my addr 172.21.15.16:0/4153633360 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:52:49.619 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:48.853+0000 7f6e8f7fe640 1 -- 172.21.15.16:0/4153633360 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e90102820 con 0x7f6e90101d90 2024-06-18T02:52:49.619 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:48.853+0000 7f6e8f7fe640 1 --2- 172.21.15.16:0/4153633360 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e90101d90 0x7f6e901021a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6e80009840 tx=0x7f6e800091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfe36ec659706ae6 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:49.619 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:48.853+0000 7f6e8e7fc640 1 -- 172.21.15.16:0/4153633360 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e80032440 con 0x7f6e90101d90 2024-06-18T02:52:49.619 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:48.853+0000 7f6e8e7fc640 1 -- 172.21.15.16:0/4153633360 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6e80032a70 con 0x7f6e90101d90 2024-06-18T02:52:49.619 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:48.853+0000 7f6e8e7fc640 1 -- 172.21.15.16:0/4153633360 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e80031ad0 con 0x7f6e90101d90 2024-06-18T02:52:49.619 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:48.854+0000 7f6e9623c640 1 -- 172.21.15.16:0/4153633360 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e90101d90 msgr2=0x7f6e901021a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:49.619 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:48.854+0000 7f6e9623c640 1 --2- 172.21.15.16:0/4153633360 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e90101d90 0x7f6e901021a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6e80009840 tx=0x7f6e800091d0 comp rx=0 tx=0).stop 2024-06-18T02:52:49.620 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:48.854+0000 7f6e9623c640 1 -- 172.21.15.16:0/4153633360 shutdown_connections 2024-06-18T02:52:49.620 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:48.854+0000 7f6e9623c640 1 --2- 172.21.15.16:0/4153633360 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e90101d90 0x7f6e901021a0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:49.620 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:48.854+0000 7f6e9623c640 1 -- 172.21.15.16:0/4153633360 >> 172.21.15.16:0/4153633360 conn(0x7f6e900fd590 msgr2=0x7f6e900ff9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:52:49.620 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:48.854+0000 7f6e9623c640 1 -- 172.21.15.16:0/4153633360 shutdown_connections 2024-06-18T02:52:49.620 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:48.854+0000 7f6e9623c640 1 -- 172.21.15.16:0/4153633360 wait complete. 2024-06-18T02:52:49.620 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:48.855+0000 7f6e9623c640 1 Processor -- start 2024-06-18T02:52:49.620 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:48.855+0000 7f6e9623c640 1 -- start start 2024-06-18T02:52:49.620 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:48.855+0000 7f6e9623c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e90101d90 0x7f6e90113080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:49.620 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:48.855+0000 7f6e9623c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e901135c0 con 0x7f6e90101d90 2024-06-18T02:52:49.620 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:48.855+0000 7f6e8f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e90101d90 0x7f6e90113080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:49.620 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:48.855+0000 7f6e8f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e90101d90 0x7f6e90113080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43074/0 (socket says 172.21.15.16:43074) 2024-06-18T02:52:49.621 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:48.855+0000 7f6e8f7fe640 1 -- 172.21.15.16:0/2663775560 learned_addr learned my addr 172.21.15.16:0/2663775560 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:52:49.621 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:48.855+0000 7f6e8f7fe640 1 -- 172.21.15.16:0/2663775560 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e80009450 con 0x7f6e90101d90 2024-06-18T02:52:49.621 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:48.856+0000 7f6e8f7fe640 1 --2- 172.21.15.16:0/2663775560 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e90101d90 0x7f6e90113080 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6e80009810 tx=0x7f6e8002c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:49.621 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:48.856+0000 7f6e87fff640 1 -- 172.21.15.16:0/2663775560 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e800328b0 con 0x7f6e90101d90 2024-06-18T02:52:49.621 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:48.856+0000 7f6e9623c640 1 -- 172.21.15.16:0/2663775560 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e901137c0 con 0x7f6e90101d90 2024-06-18T02:52:49.621 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:48.856+0000 7f6e87fff640 1 -- 172.21.15.16:0/2663775560 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6e8003b910 con 0x7f6e90101d90 2024-06-18T02:52:49.621 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:48.856+0000 7f6e87fff640 1 -- 172.21.15.16:0/2663775560 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e8003a980 con 0x7f6e90101d90 2024-06-18T02:52:49.621 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:48.856+0000 7f6e9623c640 1 -- 172.21.15.16:0/2663775560 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e90113c30 con 0x7f6e90101d90 2024-06-18T02:52:49.621 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:48.857+0000 7f6e87fff640 1 -- 172.21.15.16:0/2663775560 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f6e80039070 con 0x7f6e90101d90 2024-06-18T02:52:49.621 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:48.857+0000 7f6e87fff640 1 --2- 172.21.15.16:0/2663775560 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7f6e7c034810 0x7f6e7c036cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:49.622 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:48.857+0000 7f6e9623c640 1 -- 172.21.15.16:0/2663775560 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e9004b480 con 0x7f6e90101d90 2024-06-18T02:52:49.622 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:48.857+0000 7f6e87fff640 1 -- 172.21.15.16:0/2663775560 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f6e80066d30 con 0x7f6e90101d90 2024-06-18T02:52:49.622 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:48.857+0000 7f6e8effd640 1 --2- 172.21.15.16:0/2663775560 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7f6e7c034810 0x7f6e7c036cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:49.622 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:48.857+0000 7f6e8effd640 1 --2- 172.21.15.16:0/2663775560 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7f6e7c034810 0x7f6e7c036cc0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6e78006fd0 tx=0x7f6e78006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:49.622 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:48.862+0000 7f6e87fff640 1 -- 172.21.15.16:0/2663775560 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f6e80037080 con 0x7f6e90101d90 2024-06-18T02:52:49.622 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:49.044+0000 7f6e9623c640 1 -- 172.21.15.16:0/2663775560 --> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7f6e90103d30 con 0x7f6e7c034810 2024-06-18T02:52:49.622 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:49.048+0000 7f6e87fff640 1 -- 172.21.15.16:0/2663775560 <== mgr.14118 v2:172.21.15.16:6800/1807632468 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7f6e90103d30 con 0x7f6e7c034810 2024-06-18T02:52:49.622 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:49.050+0000 7f6e9623c640 1 -- 172.21.15.16:0/2663775560 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7f6e7c034810 msgr2=0x7f6e7c036cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:49.622 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:49.050+0000 7f6e9623c640 1 --2- 172.21.15.16:0/2663775560 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7f6e7c034810 0x7f6e7c036cc0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6e78006fd0 tx=0x7f6e78006db0 comp rx=0 tx=0).stop 2024-06-18T02:52:49.622 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:49.050+0000 7f6e9623c640 1 -- 172.21.15.16:0/2663775560 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e90101d90 msgr2=0x7f6e90113080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:49.622 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:49.050+0000 7f6e9623c640 1 --2- 172.21.15.16:0/2663775560 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e90101d90 0x7f6e90113080 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6e80009810 tx=0x7f6e8002c040 comp rx=0 tx=0).stop 2024-06-18T02:52:49.623 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:49.050+0000 7f6e9623c640 1 -- 172.21.15.16:0/2663775560 shutdown_connections 2024-06-18T02:52:49.623 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:49.050+0000 7f6e9623c640 1 --2- 172.21.15.16:0/2663775560 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7f6e7c034810 0x7f6e7c036cc0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:49.623 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:49.050+0000 7f6e9623c640 1 --2- 172.21.15.16:0/2663775560 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6e90101d90 0x7f6e90113080 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:49.623 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:49.050+0000 7f6e9623c640 1 -- 172.21.15.16:0/2663775560 >> 172.21.15.16:0/2663775560 conn(0x7f6e900fd590 msgr2=0x7f6e900fe3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:52:49.623 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:49.050+0000 7f6e9623c640 1 -- 172.21.15.16:0/2663775560 shutdown_connections 2024-06-18T02:52:49.623 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:49.050+0000 7f6e9623c640 1 -- 172.21.15.16:0/2663775560 wait complete. 2024-06-18T02:52:50.532 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:50 smithi016 ceph-mon[29214]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-06-18T02:52:50.532 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:50 smithi016 ceph-mon[29214]: Saving service alertmanager spec with placement count:1 2024-06-18T02:52:50.532 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:50 smithi016 ceph-mon[29214]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:52:50.533 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:50 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/4171684130' entity='client.admin' 2024-06-18T02:52:50.857 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:50.048+0000 7f1183aa1640 1 Processor -- start 2024-06-18T02:52:50.857 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:50.048+0000 7f1183aa1640 1 -- start start 2024-06-18T02:52:50.857 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:50.048+0000 7f1183aa1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f117c075b20 0x7f117c106dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:50.857 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:50.048+0000 7f1183aa1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f117c075f30 con 0x7f117c075b20 2024-06-18T02:52:50.857 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:50.049+0000 7f118183d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f117c075b20 0x7f117c106dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:50.857 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:50.049+0000 7f118183d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f117c075b20 0x7f117c106dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43078/0 (socket says 172.21.15.16:43078) 2024-06-18T02:52:50.857 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:50.049+0000 7f118183d640 1 -- 172.21.15.16:0/796702781 learned_addr learned my addr 172.21.15.16:0/796702781 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:52:50.858 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:50.049+0000 7f118183d640 1 -- 172.21.15.16:0/796702781 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f117c1027b0 con 0x7f117c075b20 2024-06-18T02:52:50.858 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:50.050+0000 7f118183d640 1 --2- 172.21.15.16:0/796702781 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f117c075b20 0x7f117c106dc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1174009840 tx=0x7f11740091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6be7e0ff592b7f49 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:50.858 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:50.050+0000 7f118083b640 1 -- 172.21.15.16:0/796702781 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1174032440 con 0x7f117c075b20 2024-06-18T02:52:50.858 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:50.050+0000 7f118083b640 1 -- 172.21.15.16:0/796702781 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1174032a70 con 0x7f117c075b20 2024-06-18T02:52:50.858 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:50.050+0000 7f118083b640 1 -- 172.21.15.16:0/796702781 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1174031ad0 con 0x7f117c075b20 2024-06-18T02:52:50.858 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:50.050+0000 7f1183aa1640 1 -- 172.21.15.16:0/796702781 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f117c075b20 msgr2=0x7f117c106dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:50.858 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:50.050+0000 7f1183aa1640 1 --2- 172.21.15.16:0/796702781 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f117c075b20 0x7f117c106dc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1174009840 tx=0x7f11740091d0 comp rx=0 tx=0).stop 2024-06-18T02:52:50.858 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:50.050+0000 7f1183aa1640 1 -- 172.21.15.16:0/796702781 shutdown_connections 2024-06-18T02:52:50.858 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:50.050+0000 7f1183aa1640 1 --2- 172.21.15.16:0/796702781 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f117c075b20 0x7f117c106dc0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:50.858 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:50.050+0000 7f1183aa1640 1 -- 172.21.15.16:0/796702781 >> 172.21.15.16:0/796702781 conn(0x7f117c0fc430 msgr2=0x7f117c0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:52:50.858 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:50.051+0000 7f1183aa1640 1 -- 172.21.15.16:0/796702781 shutdown_connections 2024-06-18T02:52:50.858 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:50.051+0000 7f1183aa1640 1 -- 172.21.15.16:0/796702781 wait complete. 2024-06-18T02:52:50.858 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:50.051+0000 7f1183aa1640 1 Processor -- start 2024-06-18T02:52:50.859 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:50.051+0000 7f1183aa1640 1 -- start start 2024-06-18T02:52:50.859 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:50.051+0000 7f1183aa1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f117c075b20 0x7f117c112de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:50.859 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:50.051+0000 7f1183aa1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f117c113320 con 0x7f117c075b20 2024-06-18T02:52:50.859 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:50.051+0000 7f118183d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f117c075b20 0x7f117c112de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:50.859 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:50.051+0000 7f118183d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f117c075b20 0x7f117c112de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43088/0 (socket says 172.21.15.16:43088) 2024-06-18T02:52:50.859 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:50.051+0000 7f118183d640 1 -- 172.21.15.16:0/4171684130 learned_addr learned my addr 172.21.15.16:0/4171684130 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:52:50.859 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:50.052+0000 7f118183d640 1 -- 172.21.15.16:0/4171684130 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1174009450 con 0x7f117c075b20 2024-06-18T02:52:50.859 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:50.052+0000 7f118183d640 1 --2- 172.21.15.16:0/4171684130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f117c075b20 0x7f117c112de0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f11740098a0 tx=0x7f117402c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:50.859 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:50.052+0000 7f117a7fc640 1 -- 172.21.15.16:0/4171684130 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1174031e90 con 0x7f117c075b20 2024-06-18T02:52:50.860 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:50.052+0000 7f117a7fc640 1 -- 172.21.15.16:0/4171684130 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f117403b910 con 0x7f117c075b20 2024-06-18T02:52:50.860 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:50.052+0000 7f1183aa1640 1 -- 172.21.15.16:0/4171684130 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f117c113580 con 0x7f117c075b20 2024-06-18T02:52:50.860 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:50.052+0000 7f117a7fc640 1 -- 172.21.15.16:0/4171684130 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f117403a9b0 con 0x7f117c075b20 2024-06-18T02:52:50.860 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:50.052+0000 7f1183aa1640 1 -- 172.21.15.16:0/4171684130 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f117c113f70 con 0x7f117c075b20 2024-06-18T02:52:50.860 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:50.053+0000 7f117a7fc640 1 -- 172.21.15.16:0/4171684130 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f1174039070 con 0x7f117c075b20 2024-06-18T02:52:50.860 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:50.053+0000 7f117a7fc640 1 --2- 172.21.15.16:0/4171684130 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7f1164034860 0x7f1164036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:50.860 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:50.053+0000 7f117a7fc640 1 -- 172.21.15.16:0/4171684130 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f1174066fe0 con 0x7f117c075b20 2024-06-18T02:52:50.860 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:50.053+0000 7f118103c640 1 --2- 172.21.15.16:0/4171684130 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7f1164034860 0x7f1164036d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:50.860 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:50.053+0000 7f1183aa1640 1 -- 172.21.15.16:0/4171684130 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f117c04b480 con 0x7f117c075b20 2024-06-18T02:52:50.860 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:50.054+0000 7f118103c640 1 --2- 172.21.15.16:0/4171684130 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7f1164034860 0x7f1164036d10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f1170006fd0 tx=0x7f1170006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:50.860 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:50.058+0000 7f117a7fc640 1 -- 172.21.15.16:0/4171684130 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f1174037070 con 0x7f117c075b20 2024-06-18T02:52:50.860 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:50.232+0000 7f1183aa1640 1 -- 172.21.15.16:0/4171684130 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f117c107fd0 con 0x7f117c075b20 2024-06-18T02:52:50.860 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:50.235+0000 7f117a7fc640 1 -- 172.21.15.16:0/4171684130 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f117403ac20 con 0x7f117c075b20 2024-06-18T02:52:50.861 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:50.237+0000 7f1183aa1640 1 -- 172.21.15.16:0/4171684130 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7f1164034860 msgr2=0x7f1164036d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:50.861 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:50.237+0000 7f1183aa1640 1 --2- 172.21.15.16:0/4171684130 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7f1164034860 0x7f1164036d10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f1170006fd0 tx=0x7f1170006db0 comp rx=0 tx=0).stop 2024-06-18T02:52:50.861 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:50.237+0000 7f1183aa1640 1 -- 172.21.15.16:0/4171684130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f117c075b20 msgr2=0x7f117c112de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:50.861 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:50.237+0000 7f1183aa1640 1 --2- 172.21.15.16:0/4171684130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f117c075b20 0x7f117c112de0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f11740098a0 tx=0x7f117402c040 comp rx=0 tx=0).stop 2024-06-18T02:52:50.861 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:50.237+0000 7f1183aa1640 1 -- 172.21.15.16:0/4171684130 shutdown_connections 2024-06-18T02:52:50.861 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:50.237+0000 7f1183aa1640 1 --2- 172.21.15.16:0/4171684130 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7f1164034860 0x7f1164036d10 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:50.861 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:50.237+0000 7f1183aa1640 1 --2- 172.21.15.16:0/4171684130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f117c075b20 0x7f117c112de0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:50.861 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:50.237+0000 7f1183aa1640 1 -- 172.21.15.16:0/4171684130 >> 172.21.15.16:0/4171684130 conn(0x7f117c0fc430 msgr2=0x7f117c0fd080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:52:50.861 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:50.237+0000 7f1183aa1640 1 -- 172.21.15.16:0/4171684130 shutdown_connections 2024-06-18T02:52:50.861 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:50.237+0000 7f1183aa1640 1 -- 172.21.15.16:0/4171684130 wait complete. 2024-06-18T02:52:51.956 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:51.246+0000 7fb17a576640 1 Processor -- start 2024-06-18T02:52:51.956 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:51.246+0000 7fb17a576640 1 -- start start 2024-06-18T02:52:51.956 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:51.246+0000 7fb17a576640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb174101d90 0x7fb1741021a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:51.956 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:51.246+0000 7fb17a576640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1741026e0 con 0x7fb174101d90 2024-06-18T02:52:51.956 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:51.247+0000 7fb173fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb174101d90 0x7fb1741021a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:51.956 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:51.247+0000 7fb173fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb174101d90 0x7fb1741021a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39566/0 (socket says 172.21.15.16:39566) 2024-06-18T02:52:51.956 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:51.247+0000 7fb173fff640 1 -- 172.21.15.16:0/3982468019 learned_addr learned my addr 172.21.15.16:0/3982468019 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:52:51.956 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:51.247+0000 7fb173fff640 1 -- 172.21.15.16:0/3982468019 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb174102820 con 0x7fb174101d90 2024-06-18T02:52:51.956 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:51.248+0000 7fb173fff640 1 --2- 172.21.15.16:0/3982468019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb174101d90 0x7fb1741021a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb164009840 tx=0x7fb1640091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77da405ad406ee7d server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:51.957 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:51.248+0000 7fb172ffd640 1 -- 172.21.15.16:0/3982468019 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb164032440 con 0x7fb174101d90 2024-06-18T02:52:51.957 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:51.248+0000 7fb172ffd640 1 -- 172.21.15.16:0/3982468019 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb164032a70 con 0x7fb174101d90 2024-06-18T02:52:51.957 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:51.248+0000 7fb172ffd640 1 -- 172.21.15.16:0/3982468019 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb164031ad0 con 0x7fb174101d90 2024-06-18T02:52:51.957 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:51.249+0000 7fb17a576640 1 -- 172.21.15.16:0/3982468019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb174101d90 msgr2=0x7fb1741021a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:51.957 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:51.249+0000 7fb17a576640 1 --2- 172.21.15.16:0/3982468019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb174101d90 0x7fb1741021a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb164009840 tx=0x7fb1640091d0 comp rx=0 tx=0).stop 2024-06-18T02:52:51.957 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:51.249+0000 7fb17a576640 1 -- 172.21.15.16:0/3982468019 shutdown_connections 2024-06-18T02:52:51.957 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:51.249+0000 7fb17a576640 1 --2- 172.21.15.16:0/3982468019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb174101d90 0x7fb1741021a0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:51.957 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:51.249+0000 7fb17a576640 1 -- 172.21.15.16:0/3982468019 >> 172.21.15.16:0/3982468019 conn(0x7fb1740fd5b0 msgr2=0x7fb1740ff9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:52:51.957 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:51.249+0000 7fb17a576640 1 -- 172.21.15.16:0/3982468019 shutdown_connections 2024-06-18T02:52:51.957 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:51.249+0000 7fb17a576640 1 -- 172.21.15.16:0/3982468019 wait complete. 2024-06-18T02:52:51.957 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:51.249+0000 7fb17a576640 1 Processor -- start 2024-06-18T02:52:51.957 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:51.249+0000 7fb17a576640 1 -- start start 2024-06-18T02:52:51.957 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:51.250+0000 7fb17a576640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb174101d90 0x7fb174110ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:51.958 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:51.250+0000 7fb17a576640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb174111420 con 0x7fb174101d90 2024-06-18T02:52:51.958 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:51.250+0000 7fb173fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb174101d90 0x7fb174110ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:51.958 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:51.250+0000 7fb173fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb174101d90 0x7fb174110ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39570/0 (socket says 172.21.15.16:39570) 2024-06-18T02:52:51.958 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:51.250+0000 7fb173fff640 1 -- 172.21.15.16:0/3447762280 learned_addr learned my addr 172.21.15.16:0/3447762280 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:52:51.958 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:51.250+0000 7fb173fff640 1 -- 172.21.15.16:0/3447762280 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb164009450 con 0x7fb174101d90 2024-06-18T02:52:51.958 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:51.250+0000 7fb173fff640 1 --2- 172.21.15.16:0/3447762280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb174101d90 0x7fb174110ee0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb164006a70 tx=0x7fb16402c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:51.958 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:51.250+0000 7fb170ff9640 1 -- 172.21.15.16:0/3447762280 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb164031e90 con 0x7fb174101d90 2024-06-18T02:52:51.958 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:51.250+0000 7fb170ff9640 1 -- 172.21.15.16:0/3447762280 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb16403b910 con 0x7fb174101d90 2024-06-18T02:52:51.958 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:51.250+0000 7fb17a576640 1 -- 172.21.15.16:0/3447762280 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb174111620 con 0x7fb174101d90 2024-06-18T02:52:51.958 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:51.251+0000 7fb170ff9640 1 -- 172.21.15.16:0/3447762280 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb16403a940 con 0x7fb174101d90 2024-06-18T02:52:51.958 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:51.251+0000 7fb17a576640 1 -- 172.21.15.16:0/3447762280 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1741120d0 con 0x7fb174101d90 2024-06-18T02:52:51.958 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:51.251+0000 7fb170ff9640 1 -- 172.21.15.16:0/3447762280 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7fb164039070 con 0x7fb174101d90 2024-06-18T02:52:51.958 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:51.252+0000 7fb170ff9640 1 --2- 172.21.15.16:0/3447762280 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7fb15c034810 0x7fb15c036cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:51.959 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:51.252+0000 7fb17a576640 1 -- 172.21.15.16:0/3447762280 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb17405ead0 con 0x7fb174101d90 2024-06-18T02:52:51.959 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:51.252+0000 7fb170ff9640 1 -- 172.21.15.16:0/3447762280 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fb164066e60 con 0x7fb174101d90 2024-06-18T02:52:51.959 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:51.252+0000 7fb1737fe640 1 --2- 172.21.15.16:0/3447762280 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7fb15c034810 0x7fb15c036cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:51.959 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:51.252+0000 7fb1737fe640 1 --2- 172.21.15.16:0/3447762280 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7fb15c034810 0x7fb15c036cc0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb168006fd0 tx=0x7fb168006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:51.959 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:51.256+0000 7fb170ff9640 1 -- 172.21.15.16:0/3447762280 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fb16403f030 con 0x7fb174101d90 2024-06-18T02:52:51.959 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:51.433+0000 7fb17a576640 1 -- 172.21.15.16:0/3447762280 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7fb17404b480 con 0x7fb174101d90 2024-06-18T02:52:51.959 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:51.436+0000 7fb170ff9640 1 -- 172.21.15.16:0/3447762280 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7fb16403f070 con 0x7fb174101d90 2024-06-18T02:52:51.959 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:51.438+0000 7fb17a576640 1 -- 172.21.15.16:0/3447762280 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7fb15c034810 msgr2=0x7fb15c036cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:51.959 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:51.438+0000 7fb17a576640 1 --2- 172.21.15.16:0/3447762280 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7fb15c034810 0x7fb15c036cc0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb168006fd0 tx=0x7fb168006db0 comp rx=0 tx=0).stop 2024-06-18T02:52:51.960 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:51.438+0000 7fb17a576640 1 -- 172.21.15.16:0/3447762280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb174101d90 msgr2=0x7fb174110ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:51.960 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:51.438+0000 7fb17a576640 1 --2- 172.21.15.16:0/3447762280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb174101d90 0x7fb174110ee0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb164006a70 tx=0x7fb16402c040 comp rx=0 tx=0).stop 2024-06-18T02:52:51.960 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:51.438+0000 7fb17a576640 1 -- 172.21.15.16:0/3447762280 shutdown_connections 2024-06-18T02:52:51.960 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:51.438+0000 7fb17a576640 1 --2- 172.21.15.16:0/3447762280 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7fb15c034810 0x7fb15c036cc0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:51.960 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:51.438+0000 7fb17a576640 1 --2- 172.21.15.16:0/3447762280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb174101d90 0x7fb174110ee0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:51.960 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:51.438+0000 7fb17a576640 1 -- 172.21.15.16:0/3447762280 >> 172.21.15.16:0/3447762280 conn(0x7fb1740fd5b0 msgr2=0x7fb1740fe390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:52:51.960 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:51.439+0000 7fb17a576640 1 -- 172.21.15.16:0/3447762280 shutdown_connections 2024-06-18T02:52:51.960 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:51.439+0000 7fb17a576640 1 -- 172.21.15.16:0/3447762280 wait complete. 2024-06-18T02:52:51.960 INFO:teuthology.orchestra.run.smithi016.stdout:Enabling the dashboard module... 2024-06-18T02:52:52.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:52 smithi016 ceph-mon[29214]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:52:52.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:52 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/3447762280' entity='client.admin' 2024-06-18T02:52:53.729 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:53 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/811056911' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-06-18T02:52:54.003 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:52.345+0000 7faadfc20640 1 Processor -- start 2024-06-18T02:52:54.004 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:52.346+0000 7faadfc20640 1 -- start start 2024-06-18T02:52:54.004 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:52.346+0000 7faadfc20640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faad8077c20 0x7faad8076240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:54.004 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:52.346+0000 7faadfc20640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faad8078030 con 0x7faad8077c20 2024-06-18T02:52:54.004 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:52.346+0000 7faadd9bc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faad8077c20 0x7faad8076240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:54.004 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:52.346+0000 7faadd9bc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faad8077c20 0x7faad8076240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39580/0 (socket says 172.21.15.16:39580) 2024-06-18T02:52:54.004 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:52.346+0000 7faadd9bc640 1 -- 172.21.15.16:0/3614520400 learned_addr learned my addr 172.21.15.16:0/3614520400 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:52:54.004 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:52.346+0000 7faadd9bc640 1 -- 172.21.15.16:0/3614520400 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faad8076780 con 0x7faad8077c20 2024-06-18T02:52:54.004 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:52.347+0000 7faadd9bc640 1 --2- 172.21.15.16:0/3614520400 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faad8077c20 0x7faad8076240 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7faad4009840 tx=0x7faad40091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b23759f7d70fc18 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:54.004 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:52.347+0000 7faadc9ba640 1 -- 172.21.15.16:0/3614520400 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faad4032440 con 0x7faad8077c20 2024-06-18T02:52:54.005 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:52.347+0000 7faadc9ba640 1 -- 172.21.15.16:0/3614520400 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faad4032a70 con 0x7faad8077c20 2024-06-18T02:52:54.005 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:52.348+0000 7faadc9ba640 1 -- 172.21.15.16:0/3614520400 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faad4031ad0 con 0x7faad8077c20 2024-06-18T02:52:54.005 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:52.348+0000 7faadfc20640 1 -- 172.21.15.16:0/3614520400 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faad8077c20 msgr2=0x7faad8076240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:54.005 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:52.348+0000 7faadfc20640 1 --2- 172.21.15.16:0/3614520400 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faad8077c20 0x7faad8076240 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7faad4009840 tx=0x7faad40091d0 comp rx=0 tx=0).stop 2024-06-18T02:52:54.005 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:52.348+0000 7faadfc20640 1 -- 172.21.15.16:0/3614520400 shutdown_connections 2024-06-18T02:52:54.005 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:52.348+0000 7faadfc20640 1 --2- 172.21.15.16:0/3614520400 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faad8077c20 0x7faad8076240 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:54.005 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:52.348+0000 7faadfc20640 1 -- 172.21.15.16:0/3614520400 >> 172.21.15.16:0/3614520400 conn(0x7faad80fd5b0 msgr2=0x7faad80ff9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:52:54.005 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:52.348+0000 7faadfc20640 1 -- 172.21.15.16:0/3614520400 shutdown_connections 2024-06-18T02:52:54.005 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:52.348+0000 7faadfc20640 1 -- 172.21.15.16:0/3614520400 wait complete. 2024-06-18T02:52:54.005 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:52.349+0000 7faadfc20640 1 Processor -- start 2024-06-18T02:52:54.006 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:52.349+0000 7faadfc20640 1 -- start start 2024-06-18T02:52:54.006 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:52.349+0000 7faadfc20640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faad8077c20 0x7faad81130b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:54.006 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:52.349+0000 7faadfc20640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faad81135f0 con 0x7faad8077c20 2024-06-18T02:52:54.006 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:52.349+0000 7faadd9bc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faad8077c20 0x7faad81130b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:54.006 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:52.349+0000 7faadd9bc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faad8077c20 0x7faad81130b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39592/0 (socket says 172.21.15.16:39592) 2024-06-18T02:52:54.006 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:52.349+0000 7faadd9bc640 1 -- 172.21.15.16:0/811056911 learned_addr learned my addr 172.21.15.16:0/811056911 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:52:54.006 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:52.350+0000 7faadd9bc640 1 -- 172.21.15.16:0/811056911 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faad4009450 con 0x7faad8077c20 2024-06-18T02:52:54.006 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:52.350+0000 7faadd9bc640 1 --2- 172.21.15.16:0/811056911 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faad8077c20 0x7faad81130b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7faad40098a0 tx=0x7faad402c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:54.006 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:52.350+0000 7faace7fc640 1 -- 172.21.15.16:0/811056911 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faad4031e90 con 0x7faad8077c20 2024-06-18T02:52:54.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:52.350+0000 7faace7fc640 1 -- 172.21.15.16:0/811056911 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faad403b910 con 0x7faad8077c20 2024-06-18T02:52:54.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:52.350+0000 7faadfc20640 1 -- 172.21.15.16:0/811056911 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faad81137f0 con 0x7faad8077c20 2024-06-18T02:52:54.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:52.350+0000 7faace7fc640 1 -- 172.21.15.16:0/811056911 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faad403a9b0 con 0x7faad8077c20 2024-06-18T02:52:54.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:52.350+0000 7faadfc20640 1 -- 172.21.15.16:0/811056911 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faad8113c60 con 0x7faad8077c20 2024-06-18T02:52:54.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:52.351+0000 7faace7fc640 1 -- 172.21.15.16:0/811056911 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7faad4039070 con 0x7faad8077c20 2024-06-18T02:52:54.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:52.351+0000 7faace7fc640 1 --2- 172.21.15.16:0/811056911 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7faac4034810 0x7faac4036cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:54.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:52.351+0000 7faace7fc640 1 -- 172.21.15.16:0/811056911 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7faad4066db0 con 0x7faad8077c20 2024-06-18T02:52:54.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:52.351+0000 7faadd1bb640 1 --2- 172.21.15.16:0/811056911 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7faac4034810 0x7faac4036cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:54.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:52.351+0000 7faadfc20640 1 -- 172.21.15.16:0/811056911 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faad805ead0 con 0x7faad8077c20 2024-06-18T02:52:54.007 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:52.352+0000 7faadd1bb640 1 --2- 172.21.15.16:0/811056911 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7faac4034810 0x7faac4036cc0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7faac8006fd0 tx=0x7faac8006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:54.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:52.356+0000 7faace7fc640 1 -- 172.21.15.16:0/811056911 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7faad4037070 con 0x7faad8077c20 2024-06-18T02:52:54.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:52.580+0000 7faadfc20640 1 -- 172.21.15.16:0/811056911 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7faad804b480 con 0x7faad8077c20 2024-06-18T02:52:54.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:53.440+0000 7faace7fc640 1 -- 172.21.15.16:0/811056911 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 9) v1 ==== 41959+0+0 (secure 0 0 0) 0x7faad4042360 con 0x7faad8077c20 2024-06-18T02:52:54.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:53.441+0000 7faace7fc640 1 -- 172.21.15.16:0/811056911 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7faad4030930 con 0x7faad8077c20 2024-06-18T02:52:54.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:53.444+0000 7faadfc20640 1 -- 172.21.15.16:0/811056911 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7faac4034810 msgr2=0x7faac4036cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:54.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:53.444+0000 7faadfc20640 1 --2- 172.21.15.16:0/811056911 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7faac4034810 0x7faac4036cc0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7faac8006fd0 tx=0x7faac8006db0 comp rx=0 tx=0).stop 2024-06-18T02:52:54.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:53.445+0000 7faadfc20640 1 -- 172.21.15.16:0/811056911 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faad8077c20 msgr2=0x7faad81130b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:54.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:53.445+0000 7faadfc20640 1 --2- 172.21.15.16:0/811056911 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faad8077c20 0x7faad81130b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7faad40098a0 tx=0x7faad402c040 comp rx=0 tx=0).stop 2024-06-18T02:52:54.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:53.445+0000 7faadfc20640 1 -- 172.21.15.16:0/811056911 shutdown_connections 2024-06-18T02:52:54.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:53.445+0000 7faadfc20640 1 --2- 172.21.15.16:0/811056911 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7faac4034810 0x7faac4036cc0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:54.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:53.445+0000 7faadfc20640 1 --2- 172.21.15.16:0/811056911 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faad8077c20 0x7faad81130b0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:54.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:53.445+0000 7faadfc20640 1 -- 172.21.15.16:0/811056911 >> 172.21.15.16:0/811056911 conn(0x7faad80fd5b0 msgr2=0x7faad80fe420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:52:54.008 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:53.445+0000 7faadfc20640 1 -- 172.21.15.16:0/811056911 shutdown_connections 2024-06-18T02:52:54.009 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:53.445+0000 7faadfc20640 1 -- 172.21.15.16:0/811056911 wait complete. 2024-06-18T02:52:54.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:54 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/811056911' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-06-18T02:52:54.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:54 smithi016 ceph-mon[29214]: mgrmap e9: smithi016.amxydz(active, since 26s) 2024-06-18T02:52:55.130 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-06-18T02:52:55.136 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-06-18T02:52:55.136 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "available": true, 2024-06-18T02:52:55.136 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "active_name": "smithi016.amxydz", 2024-06-18T02:52:55.137 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-06-18T02:52:55.137 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-06-18T02:52:55.137 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:54.426+0000 7f12d4bc0640 1 Processor -- start 2024-06-18T02:52:55.137 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:54.426+0000 7f12d4bc0640 1 -- start start 2024-06-18T02:52:55.137 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:54.426+0000 7f12d4bc0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12d0075ac0 0x7f12d0104a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:55.137 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:54.426+0000 7f12d4bc0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12d0075ed0 con 0x7f12d0075ac0 2024-06-18T02:52:55.137 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:54.427+0000 7f12ce59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12d0075ac0 0x7f12d0104a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:55.137 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:54.427+0000 7f12ce59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12d0075ac0 0x7f12d0104a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39604/0 (socket says 172.21.15.16:39604) 2024-06-18T02:52:55.137 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:54.427+0000 7f12ce59c640 1 -- 172.21.15.16:0/2942425376 learned_addr learned my addr 172.21.15.16:0/2942425376 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:52:55.137 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:54.427+0000 7f12ce59c640 1 -- 172.21.15.16:0/2942425376 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12d0104f60 con 0x7f12d0075ac0 2024-06-18T02:52:55.137 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:54.428+0000 7f12ce59c640 1 --2- 172.21.15.16:0/2942425376 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12d0075ac0 0x7f12d0104a20 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f12c0009840 tx=0x7f12c00091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77951f63e02dc6d server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:55.137 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:54.428+0000 7f12cd59a640 1 -- 172.21.15.16:0/2942425376 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f12c0032440 con 0x7f12d0075ac0 2024-06-18T02:52:55.138 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:54.428+0000 7f12cd59a640 1 -- 172.21.15.16:0/2942425376 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f12c0032a70 con 0x7f12d0075ac0 2024-06-18T02:52:55.138 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:54.428+0000 7f12cd59a640 1 -- 172.21.15.16:0/2942425376 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f12c0031ad0 con 0x7f12d0075ac0 2024-06-18T02:52:55.138 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:54.429+0000 7f12d4bc0640 1 -- 172.21.15.16:0/2942425376 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12d0075ac0 msgr2=0x7f12d0104a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:55.138 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:54.429+0000 7f12d4bc0640 1 --2- 172.21.15.16:0/2942425376 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12d0075ac0 0x7f12d0104a20 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f12c0009840 tx=0x7f12c00091d0 comp rx=0 tx=0).stop 2024-06-18T02:52:55.138 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:54.429+0000 7f12d4bc0640 1 -- 172.21.15.16:0/2942425376 shutdown_connections 2024-06-18T02:52:55.138 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:54.429+0000 7f12d4bc0640 1 --2- 172.21.15.16:0/2942425376 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12d0075ac0 0x7f12d0104a20 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:55.138 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:54.429+0000 7f12d4bc0640 1 -- 172.21.15.16:0/2942425376 >> 172.21.15.16:0/2942425376 conn(0x7f12d00fc430 msgr2=0x7f12d00fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:52:55.138 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:54.429+0000 7f12d4bc0640 1 -- 172.21.15.16:0/2942425376 shutdown_connections 2024-06-18T02:52:55.138 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:54.429+0000 7f12d4bc0640 1 -- 172.21.15.16:0/2942425376 wait complete. 2024-06-18T02:52:55.138 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:54.430+0000 7f12d4bc0640 1 Processor -- start 2024-06-18T02:52:55.138 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:54.430+0000 7f12d4bc0640 1 -- start start 2024-06-18T02:52:55.138 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:54.430+0000 7f12d4bc0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12d0075ac0 0x7f12d0110bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:55.138 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:54.430+0000 7f12d4bc0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12d0111110 con 0x7f12d0075ac0 2024-06-18T02:52:55.139 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:54.430+0000 7f12ce59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12d0075ac0 0x7f12d0110bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:52:55.139 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:54.430+0000 7f12ce59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12d0075ac0 0x7f12d0110bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39610/0 (socket says 172.21.15.16:39610) 2024-06-18T02:52:55.139 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:54.430+0000 7f12ce59c640 1 -- 172.21.15.16:0/525919833 learned_addr learned my addr 172.21.15.16:0/525919833 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:52:55.139 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:54.430+0000 7f12ce59c640 1 -- 172.21.15.16:0/525919833 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12c0009450 con 0x7f12d0075ac0 2024-06-18T02:52:55.139 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:54.431+0000 7f12ce59c640 1 --2- 172.21.15.16:0/525919833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12d0075ac0 0x7f12d0110bd0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f12c00098a0 tx=0x7f12c002c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:52:55.139 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:54.431+0000 7f12beffd640 1 -- 172.21.15.16:0/525919833 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f12c0031e90 con 0x7f12d0075ac0 2024-06-18T02:52:55.139 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:54.431+0000 7f12beffd640 1 -- 172.21.15.16:0/525919833 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f12c003b910 con 0x7f12d0075ac0 2024-06-18T02:52:55.139 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:54.431+0000 7f12d4bc0640 1 -- 172.21.15.16:0/525919833 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12d0111310 con 0x7f12d0075ac0 2024-06-18T02:52:55.139 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:54.431+0000 7f12beffd640 1 -- 172.21.15.16:0/525919833 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f12c003a9b0 con 0x7f12d0075ac0 2024-06-18T02:52:55.139 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:54.431+0000 7f12d4bc0640 1 -- 172.21.15.16:0/525919833 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12d0111780 con 0x7f12d0075ac0 2024-06-18T02:52:55.139 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:54.432+0000 7f12beffd640 1 -- 172.21.15.16:0/525919833 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 9) v1 ==== 41959+0+0 (secure 0 0 0) 0x7f12c0039070 con 0x7f12d0075ac0 2024-06-18T02:52:55.139 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:54.432+0000 7f12beffd640 1 --2- 172.21.15.16:0/525919833 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7f12b80348b0 0x7f12b8036d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:52:55.139 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:54.432+0000 7f12d4bc0640 1 -- 172.21.15.16:0/525919833 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12b0005240 con 0x7f12d0075ac0 2024-06-18T02:52:55.140 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:54.432+0000 7f12beffd640 1 -- 172.21.15.16:0/525919833 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f12c0066cd0 con 0x7f12d0075ac0 2024-06-18T02:52:55.140 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:54.432+0000 7f12cdd9b640 1 -- 172.21.15.16:0/525919833 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7f12b80348b0 msgr2=0x7f12b8036d60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/1807632468 2024-06-18T02:52:55.140 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:54.432+0000 7f12cdd9b640 1 --2- 172.21.15.16:0/525919833 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7f12b80348b0 0x7f12b8036d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-18T02:52:55.140 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:54.436+0000 7f12beffd640 1 -- 172.21.15.16:0/525919833 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f12c003f080 con 0x7f12d0075ac0 2024-06-18T02:52:55.140 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:54.633+0000 7f12cdd9b640 1 -- 172.21.15.16:0/525919833 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7f12b80348b0 msgr2=0x7f12b8036d60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/1807632468 2024-06-18T02:52:55.140 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:54.633+0000 7f12cdd9b640 1 --2- 172.21.15.16:0/525919833 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7f12b80348b0 0x7f12b8036d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-18T02:52:55.140 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:54.656+0000 7f12d4bc0640 1 -- 172.21.15.16:0/525919833 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f12b0005d00 con 0x7f12d0075ac0 2024-06-18T02:52:55.140 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:54.657+0000 7f12beffd640 1 -- 172.21.15.16:0/525919833 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+103 (secure 0 0 0) 0x7f12c0037070 con 0x7f12d0075ac0 2024-06-18T02:52:55.140 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:54.659+0000 7f12d4bc0640 1 -- 172.21.15.16:0/525919833 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7f12b80348b0 msgr2=0x7f12b8036d60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-18T02:52:55.140 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:54.659+0000 7f12d4bc0640 1 --2- 172.21.15.16:0/525919833 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7f12b80348b0 0x7f12b8036d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:55.140 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:54.659+0000 7f12d4bc0640 1 -- 172.21.15.16:0/525919833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12d0075ac0 msgr2=0x7f12d0110bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:52:55.140 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:54.659+0000 7f12d4bc0640 1 --2- 172.21.15.16:0/525919833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12d0075ac0 0x7f12d0110bd0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f12c00098a0 tx=0x7f12c002c040 comp rx=0 tx=0).stop 2024-06-18T02:52:55.140 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:54.659+0000 7f12d4bc0640 1 -- 172.21.15.16:0/525919833 shutdown_connections 2024-06-18T02:52:55.141 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:54.659+0000 7f12d4bc0640 1 --2- 172.21.15.16:0/525919833 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7f12b80348b0 0x7f12b8036d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:55.141 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:54.659+0000 7f12d4bc0640 1 --2- 172.21.15.16:0/525919833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12d0075ac0 0x7f12d0110bd0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:52:55.141 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:54.659+0000 7f12d4bc0640 1 -- 172.21.15.16:0/525919833 >> 172.21.15.16:0/525919833 conn(0x7f12d00fc430 msgr2=0x7f12d00fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:52:55.141 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:54.659+0000 7f12d4bc0640 1 -- 172.21.15.16:0/525919833 shutdown_connections 2024-06-18T02:52:55.141 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:54.659+0000 7f12d4bc0640 1 -- 172.21.15.16:0/525919833 wait complete. 2024-06-18T02:52:55.141 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for the mgr to restart... 2024-06-18T02:52:55.141 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for mgr epoch 9... 2024-06-18T02:52:55.491 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:55 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/525919833' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-06-18T02:52:58.305 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:58 smithi016 ceph-mon[29214]: Active manager daemon smithi016.amxydz restarted 2024-06-18T02:52:58.305 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:58 smithi016 ceph-mon[29214]: Activating manager daemon smithi016.amxydz 2024-06-18T02:52:58.305 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:58 smithi016 ceph-mon[29214]: osdmap e3: 0 total, 0 up, 0 in 2024-06-18T02:52:58.305 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:58 smithi016 ceph-mon[29214]: mgrmap e10: smithi016.amxydz(active, starting, since 0.0029122s) 2024-06-18T02:52:58.305 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:58 smithi016 ceph-mon[29214]: from='mgr.14160 172.21.15.16:0/3988449102' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "mon metadata", "id": "smithi016"}]: dispatch 2024-06-18T02:52:58.305 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:58 smithi016 ceph-mon[29214]: from='mgr.14160 172.21.15.16:0/3988449102' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "mgr metadata", "who": "smithi016.amxydz", "id": "smithi016.amxydz"}]: dispatch 2024-06-18T02:52:58.305 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:58 smithi016 ceph-mon[29214]: from='mgr.14160 172.21.15.16:0/3988449102' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-18T02:52:58.305 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:58 smithi016 ceph-mon[29214]: from='mgr.14160 172.21.15.16:0/3988449102' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-18T02:52:58.305 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:58 smithi016 ceph-mon[29214]: from='mgr.14160 172.21.15.16:0/3988449102' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-18T02:52:58.306 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:58 smithi016 ceph-mon[29214]: Manager daemon smithi016.amxydz is now available 2024-06-18T02:52:58.306 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:58 smithi016 ceph-mon[29214]: from='mgr.14160 172.21.15.16:0/3988449102' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:52:58.306 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:58 smithi016 ceph-mon[29214]: from='mgr.14160 172.21.15.16:0/3988449102' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:52:58.306 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:58 smithi016 ceph-mon[29214]: from='mgr.14160 172.21.15.16:0/3988449102' entity='mgr.smithi016.amxydz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi016.amxydz/mirror_snapshot_schedule"}]: dispatch 2024-06-18T02:52:58.306 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:52:58 smithi016 ceph-mon[29214]: from='mgr.14160 172.21.15.16:0/3988449102' entity='mgr.smithi016.amxydz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi016.amxydz/trash_purge_schedule"}]: dispatch 2024-06-18T02:53:00.071 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-06-18T02:53:00.072 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-06-18T02:53:00.072 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "initialized": true 2024-06-18T02:53:00.072 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-06-18T02:53:00.072 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:55.574+0000 7fa4d80a1640 1 Processor -- start 2024-06-18T02:53:00.072 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:55.575+0000 7fa4d80a1640 1 -- start start 2024-06-18T02:53:00.072 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:55.575+0000 7fa4d80a1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4d0075ba0 0x7fa4d0104940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:53:00.072 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:55.575+0000 7fa4d80a1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d0104e80 con 0x7fa4d0075ba0 2024-06-18T02:53:00.072 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:55.575+0000 7fa4d5e3d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4d0075ba0 0x7fa4d0104940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:53:00.072 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:55.575+0000 7fa4d5e3d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4d0075ba0 0x7fa4d0104940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39624/0 (socket says 172.21.15.16:39624) 2024-06-18T02:53:00.072 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:55.575+0000 7fa4d5e3d640 1 -- 172.21.15.16:0/133601636 learned_addr learned my addr 172.21.15.16:0/133601636 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:53:00.072 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:55.575+0000 7fa4d5e3d640 1 -- 172.21.15.16:0/133601636 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4d0104fc0 con 0x7fa4d0075ba0 2024-06-18T02:53:00.072 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:55.576+0000 7fa4d5e3d640 1 --2- 172.21.15.16:0/133601636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4d0075ba0 0x7fa4d0104940 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa4cc009840 tx=0x7fa4cc0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75f94297272245e7 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:53:00.073 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:55.576+0000 7fa4d4e3b640 1 -- 172.21.15.16:0/133601636 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4cc032440 con 0x7fa4d0075ba0 2024-06-18T02:53:00.073 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:55.576+0000 7fa4d4e3b640 1 -- 172.21.15.16:0/133601636 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa4cc032a70 con 0x7fa4d0075ba0 2024-06-18T02:53:00.073 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:55.576+0000 7fa4d4e3b640 1 -- 172.21.15.16:0/133601636 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4cc031ad0 con 0x7fa4d0075ba0 2024-06-18T02:53:00.073 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:55.577+0000 7fa4d80a1640 1 -- 172.21.15.16:0/133601636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4d0075ba0 msgr2=0x7fa4d0104940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:53:00.073 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:55.577+0000 7fa4d80a1640 1 --2- 172.21.15.16:0/133601636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4d0075ba0 0x7fa4d0104940 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa4cc009840 tx=0x7fa4cc0091d0 comp rx=0 tx=0).stop 2024-06-18T02:53:00.073 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:55.577+0000 7fa4d80a1640 1 -- 172.21.15.16:0/133601636 shutdown_connections 2024-06-18T02:53:00.073 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:55.577+0000 7fa4d80a1640 1 --2- 172.21.15.16:0/133601636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4d0075ba0 0x7fa4d0104940 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:53:00.073 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:55.577+0000 7fa4d80a1640 1 -- 172.21.15.16:0/133601636 >> 172.21.15.16:0/133601636 conn(0x7fa4d00fc430 msgr2=0x7fa4d00fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:53:00.073 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:55.577+0000 7fa4d80a1640 1 -- 172.21.15.16:0/133601636 shutdown_connections 2024-06-18T02:53:00.073 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:55.577+0000 7fa4d80a1640 1 -- 172.21.15.16:0/133601636 wait complete. 2024-06-18T02:53:00.073 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:55.577+0000 7fa4d80a1640 1 Processor -- start 2024-06-18T02:53:00.073 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:55.577+0000 7fa4d80a1640 1 -- start start 2024-06-18T02:53:00.073 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:55.578+0000 7fa4d80a1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4d0075ba0 0x7fa4d0110be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:53:00.074 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:55.578+0000 7fa4d80a1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d0111120 con 0x7fa4d0075ba0 2024-06-18T02:53:00.074 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:55.578+0000 7fa4d5e3d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4d0075ba0 0x7fa4d0110be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:53:00.074 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:55.578+0000 7fa4d5e3d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4d0075ba0 0x7fa4d0110be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39630/0 (socket says 172.21.15.16:39630) 2024-06-18T02:53:00.074 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:55.578+0000 7fa4d5e3d640 1 -- 172.21.15.16:0/3135236431 learned_addr learned my addr 172.21.15.16:0/3135236431 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:53:00.074 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:55.578+0000 7fa4d5e3d640 1 -- 172.21.15.16:0/3135236431 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4cc009450 con 0x7fa4d0075ba0 2024-06-18T02:53:00.074 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:55.578+0000 7fa4d5e3d640 1 --2- 172.21.15.16:0/3135236431 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4d0075ba0 0x7fa4d0110be0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa4cc0317f0 tx=0x7fa4cc02c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:53:00.074 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:55.578+0000 7fa4c67fc640 1 -- 172.21.15.16:0/3135236431 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4cc032900 con 0x7fa4d0075ba0 2024-06-18T02:53:00.074 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:55.578+0000 7fa4c67fc640 1 -- 172.21.15.16:0/3135236431 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa4cc03b910 con 0x7fa4d0075ba0 2024-06-18T02:53:00.074 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:55.578+0000 7fa4d80a1640 1 -- 172.21.15.16:0/3135236431 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4d0111320 con 0x7fa4d0075ba0 2024-06-18T02:53:00.074 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:55.578+0000 7fa4c67fc640 1 -- 172.21.15.16:0/3135236431 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4cc03a960 con 0x7fa4d0075ba0 2024-06-18T02:53:00.074 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:55.578+0000 7fa4d80a1640 1 -- 172.21.15.16:0/3135236431 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4d0111790 con 0x7fa4d0075ba0 2024-06-18T02:53:00.074 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:55.579+0000 7fa4c67fc640 1 -- 172.21.15.16:0/3135236431 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 9) v1 ==== 41959+0+0 (secure 0 0 0) 0x7fa4cc039070 con 0x7fa4d0075ba0 2024-06-18T02:53:00.074 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:55.579+0000 7fa4c67fc640 1 --2- 172.21.15.16:0/3135236431 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7fa4bc034860 0x7fa4bc036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:53:00.075 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:55.579+0000 7fa4c67fc640 1 -- 172.21.15.16:0/3135236431 --> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa4cc032900 con 0x7fa4bc034860 2024-06-18T02:53:00.075 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:55.579+0000 7fa4c67fc640 1 -- 172.21.15.16:0/3135236431 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fa4cc067040 con 0x7fa4d0075ba0 2024-06-18T02:53:00.075 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:55.579+0000 7fa4d563c640 1 -- 172.21.15.16:0/3135236431 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7fa4bc034860 msgr2=0x7fa4bc036d10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/1807632468 2024-06-18T02:53:00.075 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:55.580+0000 7fa4d563c640 1 --2- 172.21.15.16:0/3135236431 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7fa4bc034860 0x7fa4bc036d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-18T02:53:00.075 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:55.781+0000 7fa4d563c640 1 -- 172.21.15.16:0/3135236431 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7fa4bc034860 msgr2=0x7fa4bc036d10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/1807632468 2024-06-18T02:53:00.075 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:55.781+0000 7fa4d563c640 1 --2- 172.21.15.16:0/3135236431 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7fa4bc034860 0x7fa4bc036d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-18T02:53:00.075 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:56.182+0000 7fa4d563c640 1 -- 172.21.15.16:0/3135236431 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7fa4bc034860 msgr2=0x7fa4bc036d10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/1807632468 2024-06-18T02:53:00.075 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:56.182+0000 7fa4d563c640 1 --2- 172.21.15.16:0/3135236431 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7fa4bc034860 0x7fa4bc036d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-18T02:53:00.075 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:56.983+0000 7fa4d563c640 1 -- 172.21.15.16:0/3135236431 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7fa4bc034860 msgr2=0x7fa4bc036d10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/1807632468 2024-06-18T02:53:00.075 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:56.983+0000 7fa4d563c640 1 --2- 172.21.15.16:0/3135236431 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7fa4bc034860 0x7fa4bc036d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-18T02:53:00.075 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:58.170+0000 7fa4c67fc640 1 -- 172.21.15.16:0/3135236431 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mgrmap(e 10) v1 ==== 41726+0+0 (secure 0 0 0) 0x7fa4cc0661e0 con 0x7fa4d0075ba0 2024-06-18T02:53:00.075 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:58.170+0000 7fa4c67fc640 1 -- 172.21.15.16:0/3135236431 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7fa4bc034860 msgr2=0x7fa4bc036d10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-18T02:53:00.075 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:58.170+0000 7fa4c67fc640 1 --2- 172.21.15.16:0/3135236431 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7fa4bc034860 0x7fa4bc036d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:53:00.076 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:59.171+0000 7fa4c67fc640 1 -- 172.21.15.16:0/3135236431 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 11) v1 ==== 41853+0+0 (secure 0 0 0) 0x7fa4cc068380 con 0x7fa4d0075ba0 2024-06-18T02:53:00.076 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:59.171+0000 7fa4c67fc640 1 --2- 172.21.15.16:0/3135236431 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7fa4bc037fd0 0x7fa4bc03a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:53:00.076 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:59.171+0000 7fa4c67fc640 1 -- 172.21.15.16:0/3135236431 --> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa4cc032900 con 0x7fa4bc037fd0 2024-06-18T02:53:00.076 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:59.172+0000 7fa4d563c640 1 --2- 172.21.15.16:0/3135236431 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7fa4bc037fd0 0x7fa4bc03a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:53:00.076 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:59.173+0000 7fa4d563c640 1 --2- 172.21.15.16:0/3135236431 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7fa4bc037fd0 0x7fa4bc03a3b0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fa4c0003940 tx=0x7fa4c0009350 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:53:00.076 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:59.173+0000 7fa4c67fc640 1 -- 172.21.15.16:0/3135236431 <== mgr.14160 v2:172.21.15.16:6800/3749129160 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7fa4cc032900 con 0x7fa4bc037fd0 2024-06-18T02:53:00.076 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:59.178+0000 7fa4d80a1640 1 -- 172.21.15.16:0/3135236431 --> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fa4d004b480 con 0x7fa4bc037fd0 2024-06-18T02:53:00.076 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:59.179+0000 7fa4c67fc640 1 -- 172.21.15.16:0/3135236431 <== mgr.14160 v2:172.21.15.16:6800/3749129160 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7fa4d004b480 con 0x7fa4bc037fd0 2024-06-18T02:53:00.076 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:59.179+0000 7fa4d80a1640 1 -- 172.21.15.16:0/3135236431 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7fa4bc037fd0 msgr2=0x7fa4bc03a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:53:00.076 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:59.179+0000 7fa4d80a1640 1 --2- 172.21.15.16:0/3135236431 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7fa4bc037fd0 0x7fa4bc03a3b0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fa4c0003940 tx=0x7fa4c0009350 comp rx=0 tx=0).stop 2024-06-18T02:53:00.076 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:59.179+0000 7fa4d80a1640 1 -- 172.21.15.16:0/3135236431 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4d0075ba0 msgr2=0x7fa4d0110be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:53:00.076 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:59.179+0000 7fa4d80a1640 1 --2- 172.21.15.16:0/3135236431 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4d0075ba0 0x7fa4d0110be0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa4cc0317f0 tx=0x7fa4cc02c040 comp rx=0 tx=0).stop 2024-06-18T02:53:00.076 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:59.179+0000 7fa4d80a1640 1 -- 172.21.15.16:0/3135236431 shutdown_connections 2024-06-18T02:53:00.076 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:59.179+0000 7fa4d80a1640 1 --2- 172.21.15.16:0/3135236431 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7fa4bc037fd0 0x7fa4bc03a3b0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:53:00.077 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:59.179+0000 7fa4d80a1640 1 --2- 172.21.15.16:0/3135236431 >> [v2:172.21.15.16:6800/1807632468,v1:172.21.15.16:6801/1807632468] conn(0x7fa4bc034860 0x7fa4bc036d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:53:00.077 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:59.179+0000 7fa4d80a1640 1 --2- 172.21.15.16:0/3135236431 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4d0075ba0 0x7fa4d0110be0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:53:00.077 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:59.179+0000 7fa4d80a1640 1 -- 172.21.15.16:0/3135236431 >> 172.21.15.16:0/3135236431 conn(0x7fa4d00fc430 msgr2=0x7fa4d00fd120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:53:00.077 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:59.179+0000 7fa4d80a1640 1 -- 172.21.15.16:0/3135236431 shutdown_connections 2024-06-18T02:53:00.077 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:52:59.179+0000 7fa4d80a1640 1 -- 172.21.15.16:0/3135236431 wait complete. 2024-06-18T02:53:00.077 INFO:teuthology.orchestra.run.smithi016.stdout:mgr epoch 9 is available 2024-06-18T02:53:00.077 INFO:teuthology.orchestra.run.smithi016.stdout:Generating a dashboard self-signed certificate... 2024-06-18T02:53:00.374 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:00 smithi016 ceph-mon[29214]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-06-18T02:53:00.374 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:00 smithi016 ceph-mon[29214]: mgrmap e11: smithi016.amxydz(active, since 1.00556s) 2024-06-18T02:53:00.374 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:00 smithi016 ceph-mon[29214]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-06-18T02:53:00.374 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:00 smithi016 ceph-mon[29214]: from='mgr.14160 172.21.15.16:0/3988449102' entity='mgr.smithi016.amxydz' 2024-06-18T02:53:00.374 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:00 smithi016 ceph-mon[29214]: from='mgr.14160 172.21.15.16:0/3988449102' entity='mgr.smithi016.amxydz' 2024-06-18T02:53:01.318 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-06-18T02:53:01.319 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:00.451+0000 7fa844b26640 1 Processor -- start 2024-06-18T02:53:01.319 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:00.451+0000 7fa844b26640 1 -- start start 2024-06-18T02:53:01.319 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:00.452+0000 7fa844b26640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa840100c50 0x7fa840101060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:53:01.319 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:00.452+0000 7fa844b26640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8401015a0 con 0x7fa840100c50 2024-06-18T02:53:01.319 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:00.452+0000 7fa83e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa840100c50 0x7fa840101060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:53:01.319 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:00.452+0000 7fa83e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa840100c50 0x7fa840101060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39716/0 (socket says 172.21.15.16:39716) 2024-06-18T02:53:01.319 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:00.452+0000 7fa83e59c640 1 -- 172.21.15.16:0/3175468025 learned_addr learned my addr 172.21.15.16:0/3175468025 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:53:01.319 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:00.452+0000 7fa83e59c640 1 -- 172.21.15.16:0/3175468025 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8401016e0 con 0x7fa840100c50 2024-06-18T02:53:01.319 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:00.453+0000 7fa83e59c640 1 --2- 172.21.15.16:0/3175468025 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa840100c50 0x7fa840101060 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa830009840 tx=0x7fa8300091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cbe41132a4e3676 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:53:01.319 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:00.453+0000 7fa83dd9b640 1 -- 172.21.15.16:0/3175468025 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa830032440 con 0x7fa840100c50 2024-06-18T02:53:01.319 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:00.453+0000 7fa83dd9b640 1 -- 172.21.15.16:0/3175468025 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa830032a70 con 0x7fa840100c50 2024-06-18T02:53:01.319 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:00.453+0000 7fa83dd9b640 1 -- 172.21.15.16:0/3175468025 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa830031ad0 con 0x7fa840100c50 2024-06-18T02:53:01.320 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:00.454+0000 7fa844b26640 1 -- 172.21.15.16:0/3175468025 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa840100c50 msgr2=0x7fa840101060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:53:01.320 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:00.454+0000 7fa844b26640 1 --2- 172.21.15.16:0/3175468025 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa840100c50 0x7fa840101060 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa830009840 tx=0x7fa8300091d0 comp rx=0 tx=0).stop 2024-06-18T02:53:01.320 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:00.454+0000 7fa844b26640 1 -- 172.21.15.16:0/3175468025 shutdown_connections 2024-06-18T02:53:01.320 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:00.454+0000 7fa844b26640 1 --2- 172.21.15.16:0/3175468025 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa840100c50 0x7fa840101060 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:53:01.320 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:00.454+0000 7fa844b26640 1 -- 172.21.15.16:0/3175468025 >> 172.21.15.16:0/3175468025 conn(0x7fa8400fc430 msgr2=0x7fa8400fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:53:01.320 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:00.454+0000 7fa844b26640 1 -- 172.21.15.16:0/3175468025 shutdown_connections 2024-06-18T02:53:01.320 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:00.454+0000 7fa844b26640 1 -- 172.21.15.16:0/3175468025 wait complete. 2024-06-18T02:53:01.320 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:00.455+0000 7fa844b26640 1 Processor -- start 2024-06-18T02:53:01.320 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:00.455+0000 7fa844b26640 1 -- start start 2024-06-18T02:53:01.320 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:00.455+0000 7fa844b26640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa840100c50 0x7fa840110bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:53:01.320 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:00.455+0000 7fa844b26640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa840111110 con 0x7fa840100c50 2024-06-18T02:53:01.320 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:00.455+0000 7fa83e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa840100c50 0x7fa840110bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:53:01.320 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:00.455+0000 7fa83e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa840100c50 0x7fa840110bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39730/0 (socket says 172.21.15.16:39730) 2024-06-18T02:53:01.320 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:00.455+0000 7fa83e59c640 1 -- 172.21.15.16:0/3471447900 learned_addr learned my addr 172.21.15.16:0/3471447900 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:53:01.321 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:00.455+0000 7fa83e59c640 1 -- 172.21.15.16:0/3471447900 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa830009450 con 0x7fa840100c50 2024-06-18T02:53:01.321 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:00.455+0000 7fa83e59c640 1 --2- 172.21.15.16:0/3471447900 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa840100c50 0x7fa840110bd0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa830006a70 tx=0x7fa83002c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:53:01.321 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:00.456+0000 7fa836ffd640 1 -- 172.21.15.16:0/3471447900 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa830031e90 con 0x7fa840100c50 2024-06-18T02:53:01.321 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:00.456+0000 7fa836ffd640 1 -- 172.21.15.16:0/3471447900 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa83003b910 con 0x7fa840100c50 2024-06-18T02:53:01.321 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:00.456+0000 7fa844b26640 1 -- 172.21.15.16:0/3471447900 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa840111310 con 0x7fa840100c50 2024-06-18T02:53:01.321 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:00.456+0000 7fa836ffd640 1 -- 172.21.15.16:0/3471447900 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa83003a940 con 0x7fa840100c50 2024-06-18T02:53:01.321 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:00.456+0000 7fa844b26640 1 -- 172.21.15.16:0/3471447900 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa840111780 con 0x7fa840100c50 2024-06-18T02:53:01.321 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:00.457+0000 7fa836ffd640 1 -- 172.21.15.16:0/3471447900 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 11) v1 ==== 41853+0+0 (secure 0 0 0) 0x7fa830039070 con 0x7fa840100c50 2024-06-18T02:53:01.321 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:00.457+0000 7fa844b26640 1 -- 172.21.15.16:0/3471447900 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa84004b480 con 0x7fa840100c50 2024-06-18T02:53:01.321 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:00.457+0000 7fa836ffd640 1 --2- 172.21.15.16:0/3471447900 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7fa82c034790 0x7fa82c036c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:53:01.321 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:00.457+0000 7fa836ffd640 1 -- 172.21.15.16:0/3471447900 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fa830066ca0 con 0x7fa840100c50 2024-06-18T02:53:01.321 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:00.457+0000 7fa837fff640 1 --2- 172.21.15.16:0/3471447900 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7fa82c034790 0x7fa82c036c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:53:01.321 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:00.458+0000 7fa837fff640 1 --2- 172.21.15.16:0/3471447900 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7fa82c034790 0x7fa82c036c40 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa828006fd0 tx=0x7fa828006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:53:01.322 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:00.466+0000 7fa836ffd640 1 -- 172.21.15.16:0/3471447900 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fa830030750 con 0x7fa840100c50 2024-06-18T02:53:01.322 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:00.647+0000 7fa844b26640 1 -- 172.21.15.16:0/3471447900 --> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7fa840111780 con 0x7fa82c034790 2024-06-18T02:53:01.322 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:00.659+0000 7fa836ffd640 1 -- 172.21.15.16:0/3471447900 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mgrmap(e 12) v1 ==== 41959+0+0 (secure 0 0 0) 0x7fa83003ada0 con 0x7fa840100c50 2024-06-18T02:53:01.322 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:00.716+0000 7fa836ffd640 1 -- 172.21.15.16:0/3471447900 <== mgr.14160 v2:172.21.15.16:6800/3749129160 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7fa840111780 con 0x7fa82c034790 2024-06-18T02:53:01.322 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:00.718+0000 7fa844b26640 1 -- 172.21.15.16:0/3471447900 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7fa82c034790 msgr2=0x7fa82c036c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:53:01.322 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:00.718+0000 7fa844b26640 1 --2- 172.21.15.16:0/3471447900 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7fa82c034790 0x7fa82c036c40 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa828006fd0 tx=0x7fa828006db0 comp rx=0 tx=0).stop 2024-06-18T02:53:01.322 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:00.718+0000 7fa844b26640 1 -- 172.21.15.16:0/3471447900 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa840100c50 msgr2=0x7fa840110bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:53:01.322 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:00.718+0000 7fa844b26640 1 --2- 172.21.15.16:0/3471447900 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa840100c50 0x7fa840110bd0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa830006a70 tx=0x7fa83002c040 comp rx=0 tx=0).stop 2024-06-18T02:53:01.322 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:00.718+0000 7fa844b26640 1 -- 172.21.15.16:0/3471447900 shutdown_connections 2024-06-18T02:53:01.322 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:00.718+0000 7fa844b26640 1 --2- 172.21.15.16:0/3471447900 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7fa82c034790 0x7fa82c036c40 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:53:01.322 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:00.718+0000 7fa844b26640 1 --2- 172.21.15.16:0/3471447900 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa840100c50 0x7fa840110bd0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:53:01.322 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:00.718+0000 7fa844b26640 1 -- 172.21.15.16:0/3471447900 >> 172.21.15.16:0/3471447900 conn(0x7fa8400fc430 msgr2=0x7fa8400fd060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:53:01.322 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:00.718+0000 7fa844b26640 1 -- 172.21.15.16:0/3471447900 shutdown_connections 2024-06-18T02:53:01.323 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:00.718+0000 7fa844b26640 1 -- 172.21.15.16:0/3471447900 wait complete. 2024-06-18T02:53:01.323 INFO:teuthology.orchestra.run.smithi016.stdout:Creating initial admin user... 2024-06-18T02:53:01.442 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:01 smithi016 ceph-mon[29214]: [18/Jun/2024:02:52:59] ENGINE Bus STARTING 2024-06-18T02:53:01.442 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:01 smithi016 ceph-mon[29214]: [18/Jun/2024:02:52:59] ENGINE Serving on https://172.21.15.16:7150 2024-06-18T02:53:01.442 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:01 smithi016 ceph-mon[29214]: [18/Jun/2024:02:52:59] ENGINE Bus STARTED 2024-06-18T02:53:01.442 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:01 smithi016 ceph-mon[29214]: [18/Jun/2024:02:52:59] ENGINE Error in HTTPServer.serve 2024-06-18T02:53:01.442 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: Traceback (most recent call last): 2024-06-18T02:53:01.442 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-06-18T02:53:01.442 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: self._connections.run(self.expiration_interval) 2024-06-18T02:53:01.443 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-06-18T02:53:01.443 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: self._run(expiration_interval) 2024-06-18T02:53:01.443 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-06-18T02:53:01.443 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-06-18T02:53:01.443 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-06-18T02:53:01.443 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-06-18T02:53:01.443 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-06-18T02:53:01.443 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: s = self.context.wrap_socket( 2024-06-18T02:53:01.443 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-06-18T02:53:01.443 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: return self.sslsocket_class._create( 2024-06-18T02:53:01.443 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-06-18T02:53:01.443 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: self.do_handshake() 2024-06-18T02:53:01.443 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-06-18T02:53:01.443 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: self._sslobj.do_handshake() 2024-06-18T02:53:01.444 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1129) 2024-06-18T02:53:01.444 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:01 smithi016 ceph-mon[29214]: from='mgr.14160 172.21.15.16:0/3988449102' entity='mgr.smithi016.amxydz' 2024-06-18T02:53:01.444 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:01 smithi016 ceph-mon[29214]: mgrmap e12: smithi016.amxydz(active, since 2s) 2024-06-18T02:53:01.444 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:01 smithi016 ceph-mon[29214]: from='mgr.14160 172.21.15.16:0/3988449102' entity='mgr.smithi016.amxydz' 2024-06-18T02:53:01.444 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:01 smithi016 ceph-mon[29214]: from='mgr.14160 172.21.15.16:0/3988449102' entity='mgr.smithi016.amxydz' 2024-06-18T02:53:02.305 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:02 smithi016 ceph-mon[29214]: from='client.14172 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-06-18T02:53:02.929 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$S3ABdNofksYZDUX5.V.tKu1Am0mfSd.AfgWH2Uuu.J9RBRD64NAiG", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1718679182, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-06-18T02:53:02.930 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:01.760+0000 7f0650e32640 1 Processor -- start 2024-06-18T02:53:02.930 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:01.760+0000 7f0650e32640 1 -- start start 2024-06-18T02:53:02.930 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:01.761+0000 7f0650e32640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f064c101cd0 0x7f064c1020e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:53:02.930 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:01.761+0000 7f0650e32640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f064c102620 con 0x7f064c101cd0 2024-06-18T02:53:02.930 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:01.761+0000 7f064a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f064c101cd0 0x7f064c1020e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:53:02.930 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:01.761+0000 7f064a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f064c101cd0 0x7f064c1020e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46826/0 (socket says 172.21.15.16:46826) 2024-06-18T02:53:02.930 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:01.761+0000 7f064a59c640 1 -- 172.21.15.16:0/1271991603 learned_addr learned my addr 172.21.15.16:0/1271991603 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:53:02.930 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:01.761+0000 7f064a59c640 1 -- 172.21.15.16:0/1271991603 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f064c102760 con 0x7f064c101cd0 2024-06-18T02:53:02.930 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:01.762+0000 7f064a59c640 1 --2- 172.21.15.16:0/1271991603 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f064c101cd0 0x7f064c1020e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0644009840 tx=0x7f06440091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ae5822637f2b677 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:53:02.930 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:01.762+0000 7f064959a640 1 -- 172.21.15.16:0/1271991603 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0644032440 con 0x7f064c101cd0 2024-06-18T02:53:02.931 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:01.762+0000 7f064959a640 1 -- 172.21.15.16:0/1271991603 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0644032a70 con 0x7f064c101cd0 2024-06-18T02:53:02.931 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:01.762+0000 7f064959a640 1 -- 172.21.15.16:0/1271991603 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0644031ad0 con 0x7f064c101cd0 2024-06-18T02:53:02.931 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:01.763+0000 7f0650e32640 1 -- 172.21.15.16:0/1271991603 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f064c101cd0 msgr2=0x7f064c1020e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:53:02.931 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:01.763+0000 7f0650e32640 1 --2- 172.21.15.16:0/1271991603 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f064c101cd0 0x7f064c1020e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0644009840 tx=0x7f06440091d0 comp rx=0 tx=0).stop 2024-06-18T02:53:02.931 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:01.763+0000 7f0650e32640 1 -- 172.21.15.16:0/1271991603 shutdown_connections 2024-06-18T02:53:02.931 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:01.763+0000 7f0650e32640 1 --2- 172.21.15.16:0/1271991603 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f064c101cd0 0x7f064c1020e0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:53:02.931 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:01.763+0000 7f0650e32640 1 -- 172.21.15.16:0/1271991603 >> 172.21.15.16:0/1271991603 conn(0x7f064c0fd4b0 msgr2=0x7f064c0ff8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:53:02.931 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:01.763+0000 7f0650e32640 1 -- 172.21.15.16:0/1271991603 shutdown_connections 2024-06-18T02:53:02.931 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:01.763+0000 7f0650e32640 1 -- 172.21.15.16:0/1271991603 wait complete. 2024-06-18T02:53:02.931 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:01.763+0000 7f0650e32640 1 Processor -- start 2024-06-18T02:53:02.931 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:01.763+0000 7f0650e32640 1 -- start start 2024-06-18T02:53:02.931 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:01.764+0000 7f0650e32640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f064c101cd0 0x7f064c110b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:53:02.931 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:01.764+0000 7f0650e32640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f064c1110c0 con 0x7f064c101cd0 2024-06-18T02:53:02.931 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:01.764+0000 7f064a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f064c101cd0 0x7f064c110b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:53:02.932 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:01.764+0000 7f064a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f064c101cd0 0x7f064c110b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46836/0 (socket says 172.21.15.16:46836) 2024-06-18T02:53:02.932 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:01.764+0000 7f064a59c640 1 -- 172.21.15.16:0/2606495170 learned_addr learned my addr 172.21.15.16:0/2606495170 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:53:02.932 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:01.764+0000 7f064a59c640 1 -- 172.21.15.16:0/2606495170 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0644009450 con 0x7f064c101cd0 2024-06-18T02:53:02.932 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:01.764+0000 7f064a59c640 1 --2- 172.21.15.16:0/2606495170 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f064c101cd0 0x7f064c110b80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0644006a70 tx=0x7f064402c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:53:02.932 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:01.764+0000 7f0642ffd640 1 -- 172.21.15.16:0/2606495170 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0644031e90 con 0x7f064c101cd0 2024-06-18T02:53:02.932 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:01.764+0000 7f0650e32640 1 -- 172.21.15.16:0/2606495170 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f064c1112c0 con 0x7f064c101cd0 2024-06-18T02:53:02.932 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:01.764+0000 7f0642ffd640 1 -- 172.21.15.16:0/2606495170 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f064403b910 con 0x7f064c101cd0 2024-06-18T02:53:02.932 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:01.764+0000 7f0642ffd640 1 -- 172.21.15.16:0/2606495170 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f064403a940 con 0x7f064c101cd0 2024-06-18T02:53:02.932 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:01.764+0000 7f0650e32640 1 -- 172.21.15.16:0/2606495170 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f064c111730 con 0x7f064c101cd0 2024-06-18T02:53:02.932 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:01.765+0000 7f0642ffd640 1 -- 172.21.15.16:0/2606495170 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 12) v1 ==== 41959+0+0 (secure 0 0 0) 0x7f0644039070 con 0x7f064c101cd0 2024-06-18T02:53:02.932 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:01.766+0000 7f0642ffd640 1 --2- 172.21.15.16:0/2606495170 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f0638034860 0x7f0638036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:53:02.932 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:01.766+0000 7f0642ffd640 1 -- 172.21.15.16:0/2606495170 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f0644066f60 con 0x7f064c101cd0 2024-06-18T02:53:02.932 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:01.766+0000 7f0650e32640 1 -- 172.21.15.16:0/2606495170 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f064c05ead0 con 0x7f064c101cd0 2024-06-18T02:53:02.932 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:01.766+0000 7f0649d9b640 1 --2- 172.21.15.16:0/2606495170 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f0638034860 0x7f0638036d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:53:02.932 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:01.766+0000 7f0649d9b640 1 --2- 172.21.15.16:0/2606495170 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f0638034860 0x7f0638036d10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0634006fd0 tx=0x7f0634006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:53:02.933 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:01.770+0000 7f0642ffd640 1 -- 172.21.15.16:0/2606495170 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f0644037070 con 0x7f064c101cd0 2024-06-18T02:53:02.933 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:01.964+0000 7f0650e32640 1 -- 172.21.15.16:0/2606495170 --> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f064c04b480 con 0x7f0638034860 2024-06-18T02:53:02.933 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:02.214+0000 7f0642ffd640 1 -- 172.21.15.16:0/2606495170 <== mgr.14160 v2:172.21.15.16:6800/3749129160 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f064c04b480 con 0x7f0638034860 2024-06-18T02:53:02.933 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:02.216+0000 7f0650e32640 1 -- 172.21.15.16:0/2606495170 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f0638034860 msgr2=0x7f0638036d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:53:02.933 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:02.216+0000 7f0650e32640 1 --2- 172.21.15.16:0/2606495170 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f0638034860 0x7f0638036d10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0634006fd0 tx=0x7f0634006db0 comp rx=0 tx=0).stop 2024-06-18T02:53:02.933 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:02.216+0000 7f0650e32640 1 -- 172.21.15.16:0/2606495170 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f064c101cd0 msgr2=0x7f064c110b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:53:02.933 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:02.216+0000 7f0650e32640 1 --2- 172.21.15.16:0/2606495170 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f064c101cd0 0x7f064c110b80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0644006a70 tx=0x7f064402c040 comp rx=0 tx=0).stop 2024-06-18T02:53:02.933 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:02.216+0000 7f0650e32640 1 -- 172.21.15.16:0/2606495170 shutdown_connections 2024-06-18T02:53:02.933 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:02.216+0000 7f0650e32640 1 --2- 172.21.15.16:0/2606495170 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f0638034860 0x7f0638036d10 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:53:02.933 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:02.216+0000 7f0650e32640 1 --2- 172.21.15.16:0/2606495170 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f064c101cd0 0x7f064c110b80 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:53:02.933 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:02.216+0000 7f0650e32640 1 -- 172.21.15.16:0/2606495170 >> 172.21.15.16:0/2606495170 conn(0x7f064c0fd4b0 msgr2=0x7f064c0fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:53:02.933 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:02.216+0000 7f0650e32640 1 -- 172.21.15.16:0/2606495170 shutdown_connections 2024-06-18T02:53:02.934 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:02.216+0000 7f0650e32640 1 -- 172.21.15.16:0/2606495170 wait complete. 2024-06-18T02:53:02.934 INFO:teuthology.orchestra.run.smithi016.stdout:Fetching dashboard port number... 2024-06-18T02:53:03.488 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:03 smithi016 ceph-mon[29214]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-06-18T02:53:03.488 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:03 smithi016 ceph-mon[29214]: from='mgr.14160 172.21.15.16:0/3988449102' entity='mgr.smithi016.amxydz' 2024-06-18T02:53:04.531 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:04 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/1290134738' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-06-18T02:53:04.538 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 8443 2024-06-18T02:53:04.538 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:03.781+0000 7f1f2a99d640 1 Processor -- start 2024-06-18T02:53:04.538 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:03.782+0000 7f1f2a99d640 1 -- start start 2024-06-18T02:53:04.538 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:03.782+0000 7f1f2a99d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f24101b70 0x7f1f24101f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:53:04.538 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:03.782+0000 7f1f2a99d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f241024c0 con 0x7f1f24101b70 2024-06-18T02:53:04.539 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:03.782+0000 7f1f2999b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f24101b70 0x7f1f24101f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:53:04.539 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:03.782+0000 7f1f2999b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f24101b70 0x7f1f24101f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46838/0 (socket says 172.21.15.16:46838) 2024-06-18T02:53:04.539 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:03.782+0000 7f1f2999b640 1 -- 172.21.15.16:0/1486028514 learned_addr learned my addr 172.21.15.16:0/1486028514 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:53:04.539 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:03.782+0000 7f1f2999b640 1 -- 172.21.15.16:0/1486028514 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f24102600 con 0x7f1f24101b70 2024-06-18T02:53:04.539 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:03.783+0000 7f1f2999b640 1 --2- 172.21.15.16:0/1486028514 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f24101b70 0x7f1f24101f80 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1f20009840 tx=0x7f1f200091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e434447eb645e92 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:53:04.539 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:03.783+0000 7f1f28999640 1 -- 172.21.15.16:0/1486028514 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f20032440 con 0x7f1f24101b70 2024-06-18T02:53:04.539 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:03.783+0000 7f1f28999640 1 -- 172.21.15.16:0/1486028514 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1f20032a70 con 0x7f1f24101b70 2024-06-18T02:53:04.539 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:03.783+0000 7f1f28999640 1 -- 172.21.15.16:0/1486028514 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f20031ad0 con 0x7f1f24101b70 2024-06-18T02:53:04.539 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:03.784+0000 7f1f2a99d640 1 -- 172.21.15.16:0/1486028514 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f24101b70 msgr2=0x7f1f24101f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:53:04.539 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:03.784+0000 7f1f2a99d640 1 --2- 172.21.15.16:0/1486028514 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f24101b70 0x7f1f24101f80 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1f20009840 tx=0x7f1f200091d0 comp rx=0 tx=0).stop 2024-06-18T02:53:04.539 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:03.784+0000 7f1f2a99d640 1 -- 172.21.15.16:0/1486028514 shutdown_connections 2024-06-18T02:53:04.539 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:03.784+0000 7f1f2a99d640 1 --2- 172.21.15.16:0/1486028514 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f24101b70 0x7f1f24101f80 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:53:04.539 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:03.784+0000 7f1f2a99d640 1 -- 172.21.15.16:0/1486028514 >> 172.21.15.16:0/1486028514 conn(0x7f1f240fd330 msgr2=0x7f1f240ff760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:53:04.540 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:03.784+0000 7f1f2a99d640 1 -- 172.21.15.16:0/1486028514 shutdown_connections 2024-06-18T02:53:04.540 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:03.784+0000 7f1f2a99d640 1 -- 172.21.15.16:0/1486028514 wait complete. 2024-06-18T02:53:04.540 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:03.785+0000 7f1f2a99d640 1 Processor -- start 2024-06-18T02:53:04.540 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:03.785+0000 7f1f2a99d640 1 -- start start 2024-06-18T02:53:04.540 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:03.785+0000 7f1f2a99d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f24101b70 0x7f1f241109f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:53:04.540 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:03.785+0000 7f1f2a99d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f24110f30 con 0x7f1f24101b70 2024-06-18T02:53:04.540 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:03.785+0000 7f1f2999b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f24101b70 0x7f1f241109f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:53:04.540 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:03.785+0000 7f1f2999b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f24101b70 0x7f1f241109f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46848/0 (socket says 172.21.15.16:46848) 2024-06-18T02:53:04.540 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:03.785+0000 7f1f2999b640 1 -- 172.21.15.16:0/1290134738 learned_addr learned my addr 172.21.15.16:0/1290134738 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:53:04.540 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:03.785+0000 7f1f2999b640 1 -- 172.21.15.16:0/1290134738 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f20009450 con 0x7f1f24101b70 2024-06-18T02:53:04.540 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:03.785+0000 7f1f2999b640 1 --2- 172.21.15.16:0/1290134738 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f24101b70 0x7f1f241109f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1f20009810 tx=0x7f1f2002c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:53:04.540 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:03.785+0000 7f1f1a7fc640 1 -- 172.21.15.16:0/1290134738 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f200328b0 con 0x7f1f24101b70 2024-06-18T02:53:04.541 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:03.785+0000 7f1f1a7fc640 1 -- 172.21.15.16:0/1290134738 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1f2003b910 con 0x7f1f24101b70 2024-06-18T02:53:04.541 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:03.786+0000 7f1f2a99d640 1 -- 172.21.15.16:0/1290134738 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f24111130 con 0x7f1f24101b70 2024-06-18T02:53:04.541 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:03.786+0000 7f1f1a7fc640 1 -- 172.21.15.16:0/1290134738 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f2003a980 con 0x7f1f24101b70 2024-06-18T02:53:04.541 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:03.786+0000 7f1f2a99d640 1 -- 172.21.15.16:0/1290134738 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f241115a0 con 0x7f1f24101b70 2024-06-18T02:53:04.541 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:03.786+0000 7f1f1a7fc640 1 -- 172.21.15.16:0/1290134738 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 12) v1 ==== 41959+0+0 (secure 0 0 0) 0x7f1f20039070 con 0x7f1f24101b70 2024-06-18T02:53:04.541 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:03.787+0000 7f1f1a7fc640 1 --2- 172.21.15.16:0/1290134738 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f1f10034860 0x7f1f10036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:53:04.541 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:03.787+0000 7f1f1a7fc640 1 -- 172.21.15.16:0/1290134738 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f1f20066e10 con 0x7f1f24101b70 2024-06-18T02:53:04.541 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:03.787+0000 7f1f2a99d640 1 -- 172.21.15.16:0/1290134738 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f2405ead0 con 0x7f1f24101b70 2024-06-18T02:53:04.541 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:03.787+0000 7f1f2919a640 1 --2- 172.21.15.16:0/1290134738 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f1f10034860 0x7f1f10036d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:53:04.541 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:03.787+0000 7f1f2919a640 1 --2- 172.21.15.16:0/1290134738 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f1f10034860 0x7f1f10036d10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1f14006fd0 tx=0x7f1f14006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:53:04.541 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:03.791+0000 7f1f1a7fc640 1 -- 172.21.15.16:0/1290134738 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f1f20037070 con 0x7f1f24101b70 2024-06-18T02:53:04.541 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:03.965+0000 7f1f2a99d640 1 -- 172.21.15.16:0/1290134738 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f1f2404b480 con 0x7f1f24101b70 2024-06-18T02:53:04.541 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:03.965+0000 7f1f1a7fc640 1 -- 172.21.15.16:0/1290134738 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f1f20039500 con 0x7f1f24101b70 2024-06-18T02:53:04.542 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:03.967+0000 7f1f2a99d640 1 -- 172.21.15.16:0/1290134738 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f1f10034860 msgr2=0x7f1f10036d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:53:04.542 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:03.967+0000 7f1f2a99d640 1 --2- 172.21.15.16:0/1290134738 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f1f10034860 0x7f1f10036d10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1f14006fd0 tx=0x7f1f14006db0 comp rx=0 tx=0).stop 2024-06-18T02:53:04.542 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:03.967+0000 7f1f2a99d640 1 -- 172.21.15.16:0/1290134738 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f24101b70 msgr2=0x7f1f241109f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:53:04.542 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:03.967+0000 7f1f2a99d640 1 --2- 172.21.15.16:0/1290134738 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f24101b70 0x7f1f241109f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1f20009810 tx=0x7f1f2002c040 comp rx=0 tx=0).stop 2024-06-18T02:53:04.542 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:03.967+0000 7f1f2a99d640 1 -- 172.21.15.16:0/1290134738 shutdown_connections 2024-06-18T02:53:04.542 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:03.967+0000 7f1f2a99d640 1 --2- 172.21.15.16:0/1290134738 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f1f10034860 0x7f1f10036d10 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:53:04.542 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:03.967+0000 7f1f2a99d640 1 --2- 172.21.15.16:0/1290134738 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f24101b70 0x7f1f241109f0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:53:04.542 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:03.967+0000 7f1f2a99d640 1 -- 172.21.15.16:0/1290134738 >> 172.21.15.16:0/1290134738 conn(0x7f1f240fd330 msgr2=0x7f1f240fdef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:53:04.542 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:03.967+0000 7f1f2a99d640 1 -- 172.21.15.16:0/1290134738 shutdown_connections 2024-06-18T02:53:04.542 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:03.967+0000 7f1f2a99d640 1 -- 172.21.15.16:0/1290134738 wait complete. 2024-06-18T02:53:04.546 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-06-18T02:53:04.546 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout disabled 2024-06-18T02:53:04.554 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-06-18T02:53:04.554 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout inactive 2024-06-18T02:53:04.555 INFO:teuthology.orchestra.run.smithi016.stdout:firewalld.service is not enabled 2024-06-18T02:53:04.555 INFO:teuthology.orchestra.run.smithi016.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-06-18T02:53:04.556 INFO:teuthology.orchestra.run.smithi016.stdout:Ceph Dashboard is now available at: 2024-06-18T02:53:04.556 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:53:04.556 INFO:teuthology.orchestra.run.smithi016.stdout: URL: https://smithi016.front.sepia.ceph.com:8443/ 2024-06-18T02:53:04.557 INFO:teuthology.orchestra.run.smithi016.stdout: User: admin 2024-06-18T02:53:04.557 INFO:teuthology.orchestra.run.smithi016.stdout: Password: 3pzyzdxne8 2024-06-18T02:53:04.557 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:53:04.558 INFO:teuthology.orchestra.run.smithi016.stdout:Saving cluster configuration to /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/config directory 2024-06-18T02:53:04.558 INFO:teuthology.orchestra.run.smithi016.stdout:Enabling autotune for osd_memory_target 2024-06-18T02:53:05.412 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:05 smithi016 ceph-mon[29214]: mgrmap e13: smithi016.amxydz(active, since 6s) 2024-06-18T02:53:05.611 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:04.942+0000 7f2ff6e79640 1 Processor -- start 2024-06-18T02:53:05.611 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:04.942+0000 7f2ff6e79640 1 -- start start 2024-06-18T02:53:05.611 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:04.942+0000 7f2ff6e79640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ff0105430 0x7f2ff0105840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:53:05.611 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:04.943+0000 7f2ff6e79640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff0105e10 con 0x7f2ff0105430 2024-06-18T02:53:05.612 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:04.943+0000 7f2ff4c15640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ff0105430 0x7f2ff0105840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:53:05.612 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:04.943+0000 7f2ff4c15640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ff0105430 0x7f2ff0105840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46850/0 (socket says 172.21.15.16:46850) 2024-06-18T02:53:05.612 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:04.943+0000 7f2ff4c15640 1 -- 172.21.15.16:0/158566126 learned_addr learned my addr 172.21.15.16:0/158566126 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:53:05.612 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:04.943+0000 7f2ff4c15640 1 -- 172.21.15.16:0/158566126 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ff0105f50 con 0x7f2ff0105430 2024-06-18T02:53:05.612 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:04.944+0000 7f2ff4c15640 1 --2- 172.21.15.16:0/158566126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ff0105430 0x7f2ff0105840 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2fe0009840 tx=0x7f2fe00091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=169e7d96a414a9db server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:53:05.612 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:04.944+0000 7f2fef7fe640 1 -- 172.21.15.16:0/158566126 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2fe0032440 con 0x7f2ff0105430 2024-06-18T02:53:05.612 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:04.944+0000 7f2fef7fe640 1 -- 172.21.15.16:0/158566126 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2fe0032a70 con 0x7f2ff0105430 2024-06-18T02:53:05.612 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:04.944+0000 7f2fef7fe640 1 -- 172.21.15.16:0/158566126 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2fe0031ad0 con 0x7f2ff0105430 2024-06-18T02:53:05.612 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:04.945+0000 7f2ff6e79640 1 -- 172.21.15.16:0/158566126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ff0105430 msgr2=0x7f2ff0105840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:53:05.612 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:04.945+0000 7f2ff6e79640 1 --2- 172.21.15.16:0/158566126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ff0105430 0x7f2ff0105840 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2fe0009840 tx=0x7f2fe00091d0 comp rx=0 tx=0).stop 2024-06-18T02:53:05.612 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:04.945+0000 7f2ff6e79640 1 -- 172.21.15.16:0/158566126 shutdown_connections 2024-06-18T02:53:05.612 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:04.945+0000 7f2ff6e79640 1 --2- 172.21.15.16:0/158566126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ff0105430 0x7f2ff0105840 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:53:05.612 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:04.945+0000 7f2ff6e79640 1 -- 172.21.15.16:0/158566126 >> 172.21.15.16:0/158566126 conn(0x7f2ff00776d0 msgr2=0x7f2ff0077ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:53:05.613 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:04.945+0000 7f2ff6e79640 1 -- 172.21.15.16:0/158566126 shutdown_connections 2024-06-18T02:53:05.613 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:04.945+0000 7f2ff6e79640 1 -- 172.21.15.16:0/158566126 wait complete. 2024-06-18T02:53:05.613 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:04.945+0000 7f2ff6e79640 1 Processor -- start 2024-06-18T02:53:05.613 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:04.945+0000 7f2ff6e79640 1 -- start start 2024-06-18T02:53:05.613 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:04.946+0000 7f2ff6e79640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ff0105430 0x7f2ff0112e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:53:05.613 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:04.946+0000 7f2ff6e79640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff0113370 con 0x7f2ff0105430 2024-06-18T02:53:05.613 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:04.946+0000 7f2ff4c15640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ff0105430 0x7f2ff0112e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:53:05.613 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:04.946+0000 7f2ff4c15640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ff0105430 0x7f2ff0112e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46862/0 (socket says 172.21.15.16:46862) 2024-06-18T02:53:05.613 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:04.946+0000 7f2ff4c15640 1 -- 172.21.15.16:0/363773023 learned_addr learned my addr 172.21.15.16:0/363773023 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:53:05.613 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:04.946+0000 7f2ff4c15640 1 -- 172.21.15.16:0/363773023 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fe0009450 con 0x7f2ff0105430 2024-06-18T02:53:05.613 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:04.946+0000 7f2ff4c15640 1 --2- 172.21.15.16:0/363773023 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ff0105430 0x7f2ff0112e30 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2fe00098a0 tx=0x7f2fe0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:53:05.613 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:04.946+0000 7f2fed7fa640 1 -- 172.21.15.16:0/363773023 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2fe0031ee0 con 0x7f2ff0105430 2024-06-18T02:53:05.613 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:04.946+0000 7f2fed7fa640 1 -- 172.21.15.16:0/363773023 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2fe003d920 con 0x7f2ff0105430 2024-06-18T02:53:05.614 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:04.946+0000 7f2ff6e79640 1 -- 172.21.15.16:0/363773023 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ff0113570 con 0x7f2ff0105430 2024-06-18T02:53:05.614 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:04.946+0000 7f2fed7fa640 1 -- 172.21.15.16:0/363773023 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2fe003b9c0 con 0x7f2ff0105430 2024-06-18T02:53:05.614 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:04.947+0000 7f2ff6e79640 1 -- 172.21.15.16:0/363773023 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ff0113fc0 con 0x7f2ff0105430 2024-06-18T02:53:05.614 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:04.947+0000 7f2fed7fa640 1 -- 172.21.15.16:0/363773023 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 42002+0+0 (secure 0 0 0) 0x7f2fe0039070 con 0x7f2ff0105430 2024-06-18T02:53:05.614 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:04.948+0000 7f2fed7fa640 1 --2- 172.21.15.16:0/363773023 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f2fd0034950 0x7f2fd0036e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:53:05.614 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:04.948+0000 7f2fed7fa640 1 -- 172.21.15.16:0/363773023 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f2fe0067760 con 0x7f2ff0105430 2024-06-18T02:53:05.614 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:04.948+0000 7f2ff6e79640 1 -- 172.21.15.16:0/363773023 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ff005ead0 con 0x7f2ff0105430 2024-06-18T02:53:05.614 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:04.948+0000 7f2feffff640 1 --2- 172.21.15.16:0/363773023 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f2fd0034950 0x7f2fd0036e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:53:05.614 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:04.948+0000 7f2feffff640 1 --2- 172.21.15.16:0/363773023 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f2fd0034950 0x7f2fd0036e00 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2fe8006fd0 tx=0x7f2fe8006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:53:05.614 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:04.952+0000 7f2fed7fa640 1 -- 172.21.15.16:0/363773023 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f2fe0041020 con 0x7f2ff0105430 2024-06-18T02:53:05.614 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:05.128+0000 7f2ff6e79640 1 -- 172.21.15.16:0/363773023 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f2ff0105f50 con 0x7f2ff0105430 2024-06-18T02:53:05.614 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:05.128+0000 7f2fed7fa640 1 -- 172.21.15.16:0/363773023 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f2ff0105f50 con 0x7f2ff0105430 2024-06-18T02:53:05.614 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:05.130+0000 7f2ff6e79640 1 -- 172.21.15.16:0/363773023 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f2fd0034950 msgr2=0x7f2fd0036e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:53:05.615 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:05.130+0000 7f2ff6e79640 1 --2- 172.21.15.16:0/363773023 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f2fd0034950 0x7f2fd0036e00 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2fe8006fd0 tx=0x7f2fe8006db0 comp rx=0 tx=0).stop 2024-06-18T02:53:05.615 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:05.130+0000 7f2ff6e79640 1 -- 172.21.15.16:0/363773023 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ff0105430 msgr2=0x7f2ff0112e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:53:05.615 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:05.130+0000 7f2ff6e79640 1 --2- 172.21.15.16:0/363773023 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ff0105430 0x7f2ff0112e30 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2fe00098a0 tx=0x7f2fe0008040 comp rx=0 tx=0).stop 2024-06-18T02:53:05.615 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:05.130+0000 7f2ff6e79640 1 -- 172.21.15.16:0/363773023 shutdown_connections 2024-06-18T02:53:05.615 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:05.130+0000 7f2ff6e79640 1 --2- 172.21.15.16:0/363773023 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f2fd0034950 0x7f2fd0036e00 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:53:05.615 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:05.130+0000 7f2ff6e79640 1 --2- 172.21.15.16:0/363773023 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ff0105430 0x7f2ff0112e30 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:53:05.615 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:05.130+0000 7f2ff6e79640 1 -- 172.21.15.16:0/363773023 >> 172.21.15.16:0/363773023 conn(0x7f2ff00776d0 msgr2=0x7f2ff0101550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:53:05.615 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:05.130+0000 7f2ff6e79640 1 -- 172.21.15.16:0/363773023 shutdown_connections 2024-06-18T02:53:05.615 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:05.130+0000 7f2ff6e79640 1 -- 172.21.15.16:0/363773023 wait complete. 2024-06-18T02:53:06.835 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:05.988+0000 7fda5d1a3640 1 Processor -- start 2024-06-18T02:53:06.835 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:05.988+0000 7fda5d1a3640 1 -- start start 2024-06-18T02:53:06.835 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:05.988+0000 7fda5d1a3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda58077d10 0x7fda58076240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:53:06.836 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:05.988+0000 7fda5d1a3640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda58078120 con 0x7fda58077d10 2024-06-18T02:53:06.836 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:05.988+0000 7fda56d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda58077d10 0x7fda58076240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:53:06.836 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:05.988+0000 7fda56d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda58077d10 0x7fda58076240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46876/0 (socket says 172.21.15.16:46876) 2024-06-18T02:53:06.836 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:05.988+0000 7fda56d9d640 1 -- 172.21.15.16:0/3110429656 learned_addr learned my addr 172.21.15.16:0/3110429656 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:53:06.836 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:05.989+0000 7fda56d9d640 1 -- 172.21.15.16:0/3110429656 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda58076780 con 0x7fda58077d10 2024-06-18T02:53:06.836 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:05.989+0000 7fda56d9d640 1 --2- 172.21.15.16:0/3110429656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda58077d10 0x7fda58076240 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fda48009840 tx=0x7fda480091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34912b1dfb0789e4 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:53:06.836 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:05.989+0000 7fda5659c640 1 -- 172.21.15.16:0/3110429656 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda48032440 con 0x7fda58077d10 2024-06-18T02:53:06.836 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:05.990+0000 7fda5659c640 1 -- 172.21.15.16:0/3110429656 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fda48032a70 con 0x7fda58077d10 2024-06-18T02:53:06.836 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:05.990+0000 7fda5659c640 1 -- 172.21.15.16:0/3110429656 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda48031ad0 con 0x7fda58077d10 2024-06-18T02:53:06.836 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:05.990+0000 7fda5d1a3640 1 -- 172.21.15.16:0/3110429656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda58077d10 msgr2=0x7fda58076240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:53:06.836 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:05.990+0000 7fda5d1a3640 1 --2- 172.21.15.16:0/3110429656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda58077d10 0x7fda58076240 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fda48009840 tx=0x7fda480091d0 comp rx=0 tx=0).stop 2024-06-18T02:53:06.836 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:05.990+0000 7fda5d1a3640 1 -- 172.21.15.16:0/3110429656 shutdown_connections 2024-06-18T02:53:06.836 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:05.990+0000 7fda5d1a3640 1 --2- 172.21.15.16:0/3110429656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda58077d10 0x7fda58076240 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:53:06.837 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:05.990+0000 7fda5d1a3640 1 -- 172.21.15.16:0/3110429656 >> 172.21.15.16:0/3110429656 conn(0x7fda580fd490 msgr2=0x7fda580ff8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:53:06.837 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:05.990+0000 7fda5d1a3640 1 -- 172.21.15.16:0/3110429656 shutdown_connections 2024-06-18T02:53:06.837 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:05.990+0000 7fda5d1a3640 1 -- 172.21.15.16:0/3110429656 wait complete. 2024-06-18T02:53:06.837 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:05.991+0000 7fda5d1a3640 1 Processor -- start 2024-06-18T02:53:06.837 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:05.991+0000 7fda5d1a3640 1 -- start start 2024-06-18T02:53:06.837 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:05.991+0000 7fda5d1a3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda58077d10 0x7fda581130d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:53:06.837 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:05.991+0000 7fda5d1a3640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda58113610 con 0x7fda58077d10 2024-06-18T02:53:06.837 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:05.991+0000 7fda56d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda58077d10 0x7fda581130d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:53:06.837 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:05.991+0000 7fda56d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda58077d10 0x7fda581130d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46892/0 (socket says 172.21.15.16:46892) 2024-06-18T02:53:06.837 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:05.991+0000 7fda56d9d640 1 -- 172.21.15.16:0/2148547255 learned_addr learned my addr 172.21.15.16:0/2148547255 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:53:06.837 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:05.991+0000 7fda56d9d640 1 -- 172.21.15.16:0/2148547255 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda48009450 con 0x7fda58077d10 2024-06-18T02:53:06.837 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:05.992+0000 7fda56d9d640 1 --2- 172.21.15.16:0/2148547255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda58077d10 0x7fda581130d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fda480317f0 tx=0x7fda4802c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:53:06.838 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:05.992+0000 7fda4f7fe640 1 -- 172.21.15.16:0/2148547255 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda48032900 con 0x7fda58077d10 2024-06-18T02:53:06.838 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:05.992+0000 7fda4f7fe640 1 -- 172.21.15.16:0/2148547255 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fda4803b910 con 0x7fda58077d10 2024-06-18T02:53:06.838 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:05.992+0000 7fda4f7fe640 1 -- 172.21.15.16:0/2148547255 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda4803a960 con 0x7fda58077d10 2024-06-18T02:53:06.838 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:05.992+0000 7fda5d1a3640 1 -- 172.21.15.16:0/2148547255 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda58113810 con 0x7fda58077d10 2024-06-18T02:53:06.838 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:05.992+0000 7fda5d1a3640 1 -- 172.21.15.16:0/2148547255 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda58113c80 con 0x7fda58077d10 2024-06-18T02:53:06.838 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:05.993+0000 7fda4f7fe640 1 -- 172.21.15.16:0/2148547255 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 42002+0+0 (secure 0 0 0) 0x7fda48039070 con 0x7fda58077d10 2024-06-18T02:53:06.838 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:05.993+0000 7fda4f7fe640 1 --2- 172.21.15.16:0/2148547255 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7fda44034950 0x7fda44036e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:53:06.838 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:05.993+0000 7fda5d1a3640 1 -- 172.21.15.16:0/2148547255 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda5804b480 con 0x7fda58077d10 2024-06-18T02:53:06.838 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:05.993+0000 7fda4f7fe640 1 -- 172.21.15.16:0/2148547255 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fda48066ed0 con 0x7fda58077d10 2024-06-18T02:53:06.838 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:05.993+0000 7fda4ffff640 1 --2- 172.21.15.16:0/2148547255 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7fda44034950 0x7fda44036e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:53:06.838 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:05.993+0000 7fda4ffff640 1 --2- 172.21.15.16:0/2148547255 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7fda44034950 0x7fda44036e00 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fda40006fd0 tx=0x7fda40006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:53:06.838 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:05.998+0000 7fda4f7fe640 1 -- 172.21.15.16:0/2148547255 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fda48039500 con 0x7fda58077d10 2024-06-18T02:53:06.839 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:06.263+0000 7fda5d1a3640 1 -- 172.21.15.16:0/2148547255 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7fda58113c80 con 0x7fda58077d10 2024-06-18T02:53:06.839 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:06.265+0000 7fda4f7fe640 1 -- 172.21.15.16:0/2148547255 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v33)=0 set mgr/dashboard/cluster/status v33) v1 ==== 153+0+0 (secure 0 0 0) 0x7fda48037030 con 0x7fda58077d10 2024-06-18T02:53:06.839 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-06-18T02:53:06.839 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:06.268+0000 7fda5d1a3640 1 -- 172.21.15.16:0/2148547255 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7fda44034950 msgr2=0x7fda44036e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:53:06.839 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:06.268+0000 7fda5d1a3640 1 --2- 172.21.15.16:0/2148547255 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7fda44034950 0x7fda44036e00 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fda40006fd0 tx=0x7fda40006db0 comp rx=0 tx=0).stop 2024-06-18T02:53:06.839 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:06.268+0000 7fda5d1a3640 1 -- 172.21.15.16:0/2148547255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda58077d10 msgr2=0x7fda581130d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:53:06.839 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:06.268+0000 7fda5d1a3640 1 --2- 172.21.15.16:0/2148547255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda58077d10 0x7fda581130d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fda480317f0 tx=0x7fda4802c040 comp rx=0 tx=0).stop 2024-06-18T02:53:06.839 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:06.268+0000 7fda5d1a3640 1 -- 172.21.15.16:0/2148547255 shutdown_connections 2024-06-18T02:53:06.839 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:06.268+0000 7fda5d1a3640 1 --2- 172.21.15.16:0/2148547255 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7fda44034950 0x7fda44036e00 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:53:06.839 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:06.268+0000 7fda5d1a3640 1 --2- 172.21.15.16:0/2148547255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda58077d10 0x7fda581130d0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:53:06.839 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:06.268+0000 7fda5d1a3640 1 -- 172.21.15.16:0/2148547255 >> 172.21.15.16:0/2148547255 conn(0x7fda580fd490 msgr2=0x7fda580fe370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:53:06.839 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:06.268+0000 7fda5d1a3640 1 -- 172.21.15.16:0/2148547255 shutdown_connections 2024-06-18T02:53:06.840 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-06-18T02:53:06.268+0000 7fda5d1a3640 1 -- 172.21.15.16:0/2148547255 wait complete. 2024-06-18T02:53:06.840 INFO:teuthology.orchestra.run.smithi016.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-06-18T02:53:06.840 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:53:06.840 INFO:teuthology.orchestra.run.smithi016.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-06-18T02:53:06.840 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:53:06.840 INFO:teuthology.orchestra.run.smithi016.stdout:Or, if you are only running a single cluster on this host: 2024-06-18T02:53:06.840 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:53:06.840 INFO:teuthology.orchestra.run.smithi016.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-06-18T02:53:06.840 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:53:06.840 INFO:teuthology.orchestra.run.smithi016.stdout:Please consider enabling telemetry to help improve Ceph: 2024-06-18T02:53:06.840 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:53:06.840 INFO:teuthology.orchestra.run.smithi016.stdout: ceph telemetry on 2024-06-18T02:53:06.840 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:53:06.840 INFO:teuthology.orchestra.run.smithi016.stdout:For more information see: 2024-06-18T02:53:06.841 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:53:06.841 INFO:teuthology.orchestra.run.smithi016.stdout: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-06-18T02:53:06.841 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:53:06.841 INFO:teuthology.orchestra.run.smithi016.stdout:Bootstrap complete. 2024-06-18T02:53:06.874 INFO:tasks.cephadm:Fetching config... 2024-06-18T02:53:06.874 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-06-18T02:53:06.874 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-06-18T02:53:06.891 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-06-18T02:53:06.891 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-06-18T02:53:06.892 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-06-18T02:53:06.948 INFO:tasks.cephadm:Fetching mon keyring... 2024-06-18T02:53:06.949 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-06-18T02:53:06.949 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/keyring of=/dev/stdout 2024-06-18T02:53:07.015 INFO:tasks.cephadm:Fetching pub ssh key... 2024-06-18T02:53:07.016 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-06-18T02:53:07.016 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-06-18T02:53:07.072 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-06-18T02:53:07.072 DEBUG:teuthology.orchestra.run.smithi016:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCO/PgcQZoTguaJ90Ie1yUZnEfMoSQ3Di9RJYHMROBNTgFSMO8T1trbN1YsaouWWWDP5K3GjWENzPjALNRpFV543RB7W6ouU/eORcEsooGe7aNfuyNIEO55PA63GBvC9jTHUJ2T88QNUrAFguhQ3vENahhG9uT8F67WAiQXyliSDfF7rHdgzoVso72o+mmMudyIvg6sRyjbpLELSEtjPYId96jJ/huksO4xc2fDHmAYXvO2EgZNa0qWoYgJBAAU5UCdX0/gX1ePQoFuNKzHkeEIw0KiCwdhA+nrKRGCZ+Qc+zQ/DvvoDemgOVybCWwfu2Txtzd7G6ngHAKRZDc/5S8a7lxTHyAq8IVOylLsdhP3ZF2axRAlHpJMVVcuPh3UfDGmPNrK0WyavTqlOL7+F9jWaqXdAfZsWiC+f2dstgvo9yuGdLqdkWhF9bFsxon+ZnTGqUGLW+M8KEMOBfC//zEfMGQ+eeJPpfK/N5J9CzuOExVAPYPE+sSry0KPZ40OkGs= ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-06-18T02:53:07.154 INFO:teuthology.orchestra.run.smithi016.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCO/PgcQZoTguaJ90Ie1yUZnEfMoSQ3Di9RJYHMROBNTgFSMO8T1trbN1YsaouWWWDP5K3GjWENzPjALNRpFV543RB7W6ouU/eORcEsooGe7aNfuyNIEO55PA63GBvC9jTHUJ2T88QNUrAFguhQ3vENahhG9uT8F67WAiQXyliSDfF7rHdgzoVso72o+mmMudyIvg6sRyjbpLELSEtjPYId96jJ/huksO4xc2fDHmAYXvO2EgZNa0qWoYgJBAAU5UCdX0/gX1ePQoFuNKzHkeEIw0KiCwdhA+nrKRGCZ+Qc+zQ/DvvoDemgOVybCWwfu2Txtzd7G6ngHAKRZDc/5S8a7lxTHyAq8IVOylLsdhP3ZF2axRAlHpJMVVcuPh3UfDGmPNrK0WyavTqlOL7+F9jWaqXdAfZsWiC+f2dstgvo9yuGdLqdkWhF9bFsxon+ZnTGqUGLW+M8KEMOBfC//zEfMGQ+eeJPpfK/N5J9CzuOExVAPYPE+sSry0KPZ40OkGs= ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968 2024-06-18T02:53:07.167 DEBUG:teuthology.orchestra.run.smithi123:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCO/PgcQZoTguaJ90Ie1yUZnEfMoSQ3Di9RJYHMROBNTgFSMO8T1trbN1YsaouWWWDP5K3GjWENzPjALNRpFV543RB7W6ouU/eORcEsooGe7aNfuyNIEO55PA63GBvC9jTHUJ2T88QNUrAFguhQ3vENahhG9uT8F67WAiQXyliSDfF7rHdgzoVso72o+mmMudyIvg6sRyjbpLELSEtjPYId96jJ/huksO4xc2fDHmAYXvO2EgZNa0qWoYgJBAAU5UCdX0/gX1ePQoFuNKzHkeEIw0KiCwdhA+nrKRGCZ+Qc+zQ/DvvoDemgOVybCWwfu2Txtzd7G6ngHAKRZDc/5S8a7lxTHyAq8IVOylLsdhP3ZF2axRAlHpJMVVcuPh3UfDGmPNrK0WyavTqlOL7+F9jWaqXdAfZsWiC+f2dstgvo9yuGdLqdkWhF9bFsxon+ZnTGqUGLW+M8KEMOBfC//zEfMGQ+eeJPpfK/N5J9CzuOExVAPYPE+sSry0KPZ40OkGs= ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-06-18T02:53:07.210 INFO:teuthology.orchestra.run.smithi123.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCO/PgcQZoTguaJ90Ie1yUZnEfMoSQ3Di9RJYHMROBNTgFSMO8T1trbN1YsaouWWWDP5K3GjWENzPjALNRpFV543RB7W6ouU/eORcEsooGe7aNfuyNIEO55PA63GBvC9jTHUJ2T88QNUrAFguhQ3vENahhG9uT8F67WAiQXyliSDfF7rHdgzoVso72o+mmMudyIvg6sRyjbpLELSEtjPYId96jJ/huksO4xc2fDHmAYXvO2EgZNa0qWoYgJBAAU5UCdX0/gX1ePQoFuNKzHkeEIw0KiCwdhA+nrKRGCZ+Qc+zQ/DvvoDemgOVybCWwfu2Txtzd7G6ngHAKRZDc/5S8a7lxTHyAq8IVOylLsdhP3ZF2axRAlHpJMVVcuPh3UfDGmPNrK0WyavTqlOL7+F9jWaqXdAfZsWiC+f2dstgvo9yuGdLqdkWhF9bFsxon+ZnTGqUGLW+M8KEMOBfC//zEfMGQ+eeJPpfK/N5J9CzuOExVAPYPE+sSry0KPZ40OkGs= ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968 2024-06-18T02:53:07.223 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-06-18T02:53:07.490 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T02:53:07.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:07 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/2148547255' entity='client.admin' 2024-06-18T02:53:09.713 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:09 smithi016 ceph-mon[29214]: from='mgr.14160 172.21.15.16:0/3988449102' entity='mgr.smithi016.amxydz' 2024-06-18T02:53:09.713 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:09 smithi016 ceph-mon[29214]: from='mgr.14160 172.21.15.16:0/3988449102' entity='mgr.smithi016.amxydz' 2024-06-18T02:53:09.713 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:09 smithi016 ceph-mon[29214]: from='mgr.14160 172.21.15.16:0/3988449102' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config rm", "who": "osd/host:smithi016", "name": "osd_memory_target"}]: dispatch 2024-06-18T02:53:09.713 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:09 smithi016 ceph-mon[29214]: from='mgr.14160 172.21.15.16:0/3988449102' entity='mgr.smithi016.amxydz' 2024-06-18T02:53:09.713 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:09 smithi016 ceph-mon[29214]: from='mgr.14160 172.21.15.16:0/3988449102' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi016", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-06-18T02:53:09.714 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:09 smithi016 ceph-mon[29214]: from='mgr.14160 172.21.15.16:0/3988449102' entity='mgr.smithi016.amxydz' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi016", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-06-18T02:53:09.714 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:09 smithi016 ceph-mon[29214]: from='mgr.14160 172.21.15.16:0/3988449102' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:53:09.714 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:09 smithi016 ceph-mon[29214]: Deploying daemon crash.smithi016 on smithi016 2024-06-18T02:53:09.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:09.948+0000 7f5eca0d8640 1 -- 172.21.15.16:0/108934980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ec4065e80 msgr2=0x7f5ec40febc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:53:09.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:09.948+0000 7f5eca0d8640 1 --2- 172.21.15.16:0/108934980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ec4065e80 0x7f5ec40febc0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5eb4004850 tx=0x7f5eb4009da0 comp rx=0 tx=0).stop 2024-06-18T02:53:09.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:09.948+0000 7f5eca0d8640 1 -- 172.21.15.16:0/108934980 shutdown_connections 2024-06-18T02:53:09.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:09.948+0000 7f5eca0d8640 1 --2- 172.21.15.16:0/108934980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ec4065e80 0x7f5ec40febc0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:53:09.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:09.948+0000 7f5eca0d8640 1 -- 172.21.15.16:0/108934980 >> 172.21.15.16:0/108934980 conn(0x7f5ec40f65f0 msgr2=0x7f5ec40f8a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:53:09.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:09.948+0000 7f5eca0d8640 1 -- 172.21.15.16:0/108934980 shutdown_connections 2024-06-18T02:53:09.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:09.948+0000 7f5eca0d8640 1 -- 172.21.15.16:0/108934980 wait complete. 2024-06-18T02:53:09.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:09.949+0000 7f5eca0d8640 1 Processor -- start 2024-06-18T02:53:09.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:09.949+0000 7f5eca0d8640 1 -- start start 2024-06-18T02:53:09.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:09.949+0000 7f5eca0d8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ec4065e80 0x7f5ec410c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:53:09.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:09.949+0000 7f5eca0d8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ec410caa0 con 0x7f5ec4065e80 2024-06-18T02:53:09.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:09.949+0000 7f5ec37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ec4065e80 0x7f5ec410c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:53:09.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:09.949+0000 7f5ec37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ec4065e80 0x7f5ec410c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46920/0 (socket says 172.21.15.16:46920) 2024-06-18T02:53:09.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:09.949+0000 7f5ec37fe640 1 -- 172.21.15.16:0/2421454390 learned_addr learned my addr 172.21.15.16:0/2421454390 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:53:09.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:09.949+0000 7f5ec37fe640 1 -- 172.21.15.16:0/2421454390 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eb4005040 con 0x7f5ec4065e80 2024-06-18T02:53:09.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:09.949+0000 7f5ec37fe640 1 --2- 172.21.15.16:0/2421454390 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ec4065e80 0x7f5ec410c560 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5eb4004820 tx=0x7f5eb4009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:53:09.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:09.950+0000 7f5eabfff640 1 -- 172.21.15.16:0/2421454390 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5eb4032650 con 0x7f5ec4065e80 2024-06-18T02:53:09.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:09.950+0000 7f5eabfff640 1 -- 172.21.15.16:0/2421454390 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5eb4032c80 con 0x7f5ec4065e80 2024-06-18T02:53:09.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:09.950+0000 7f5eca0d8640 1 -- 172.21.15.16:0/2421454390 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ec410cca0 con 0x7f5ec4065e80 2024-06-18T02:53:09.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:09.950+0000 7f5eca0d8640 1 -- 172.21.15.16:0/2421454390 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ec410d830 con 0x7f5ec4065e80 2024-06-18T02:53:09.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:09.950+0000 7f5eabfff640 1 -- 172.21.15.16:0/2421454390 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5eb4033d60 con 0x7f5ec4065e80 2024-06-18T02:53:09.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:09.951+0000 7f5eabfff640 1 -- 172.21.15.16:0/2421454390 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 42002+0+0 (secure 0 0 0) 0x7f5eb4030020 con 0x7f5ec4065e80 2024-06-18T02:53:09.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:09.951+0000 7f5eabfff640 1 --2- 172.21.15.16:0/2421454390 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f5eac034950 0x7f5eac036e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:53:09.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:09.951+0000 7f5eca0d8640 1 -- 172.21.15.16:0/2421454390 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ec410d170 con 0x7f5ec4065e80 2024-06-18T02:53:09.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:09.951+0000 7f5eabfff640 1 -- 172.21.15.16:0/2421454390 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f5eb4066660 con 0x7f5ec4065e80 2024-06-18T02:53:09.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:09.951+0000 7f5ec2ffd640 1 --2- 172.21.15.16:0/2421454390 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f5eac034950 0x7f5eac036e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:53:09.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:09.952+0000 7f5ec2ffd640 1 --2- 172.21.15.16:0/2421454390 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f5eac034950 0x7f5eac036e00 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f5eb8006fd0 tx=0x7f5eb8006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:53:09.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:09.956+0000 7f5eabfff640 1 -- 172.21.15.16:0/2421454390 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f5ec410d170 con 0x7f5ec4065e80 2024-06-18T02:53:10.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:10.140+0000 7f5eca0d8640 1 -- 172.21.15.16:0/2421454390 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f5ec410d170 con 0x7f5ec4065e80 2024-06-18T02:53:10.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:10.143+0000 7f5eabfff640 1 -- 172.21.15.16:0/2421454390 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f5ec410d170 con 0x7f5ec4065e80 2024-06-18T02:53:10.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:10.145+0000 7f5eca0d8640 1 -- 172.21.15.16:0/2421454390 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f5eac034950 msgr2=0x7f5eac036e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:53:10.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:10.145+0000 7f5eca0d8640 1 --2- 172.21.15.16:0/2421454390 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f5eac034950 0x7f5eac036e00 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f5eb8006fd0 tx=0x7f5eb8006db0 comp rx=0 tx=0).stop 2024-06-18T02:53:10.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:10.145+0000 7f5eca0d8640 1 -- 172.21.15.16:0/2421454390 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ec4065e80 msgr2=0x7f5ec410c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:53:10.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:10.145+0000 7f5eca0d8640 1 --2- 172.21.15.16:0/2421454390 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ec4065e80 0x7f5ec410c560 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5eb4004820 tx=0x7f5eb4009da0 comp rx=0 tx=0).stop 2024-06-18T02:53:10.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:10.145+0000 7f5eca0d8640 1 -- 172.21.15.16:0/2421454390 shutdown_connections 2024-06-18T02:53:10.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:10.145+0000 7f5eca0d8640 1 --2- 172.21.15.16:0/2421454390 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f5eac034950 0x7f5eac036e00 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:53:10.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:10.145+0000 7f5eca0d8640 1 --2- 172.21.15.16:0/2421454390 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ec4065e80 0x7f5ec410c560 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:53:10.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:10.145+0000 7f5eca0d8640 1 -- 172.21.15.16:0/2421454390 >> 172.21.15.16:0/2421454390 conn(0x7f5ec40f65f0 msgr2=0x7f5ec41030e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:53:10.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:10.145+0000 7f5eca0d8640 1 -- 172.21.15.16:0/2421454390 shutdown_connections 2024-06-18T02:53:10.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:10.145+0000 7f5eca0d8640 1 -- 172.21.15.16:0/2421454390 wait complete. 2024-06-18T02:53:11.215 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-06-18T02:53:11.216 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-06-18T02:53:11.240 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:11 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/2421454390' entity='client.admin' 2024-06-18T02:53:12.201 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T02:53:14.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:14.127+0000 7fca7a014640 1 -- 172.21.15.16:0/4285801689 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca740653b0 msgr2=0x7fca740fed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:53:14.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:14.127+0000 7fca7a014640 1 --2- 172.21.15.16:0/4285801689 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca740653b0 0x7fca740fed50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fca64004850 tx=0x7fca64009da0 comp rx=0 tx=0).stop 2024-06-18T02:53:14.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:14.127+0000 7fca7a014640 1 -- 172.21.15.16:0/4285801689 shutdown_connections 2024-06-18T02:53:14.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:14.127+0000 7fca7a014640 1 --2- 172.21.15.16:0/4285801689 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca740653b0 0x7fca740fed50 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:53:14.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:14.127+0000 7fca7a014640 1 -- 172.21.15.16:0/4285801689 >> 172.21.15.16:0/4285801689 conn(0x7fca740f6820 msgr2=0x7fca740f8c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:53:14.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:14.127+0000 7fca7a014640 1 -- 172.21.15.16:0/4285801689 shutdown_connections 2024-06-18T02:53:14.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:14.127+0000 7fca7a014640 1 -- 172.21.15.16:0/4285801689 wait complete. 2024-06-18T02:53:14.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:14.127+0000 7fca7a014640 1 Processor -- start 2024-06-18T02:53:14.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:14.128+0000 7fca7a014640 1 -- start start 2024-06-18T02:53:14.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:14.128+0000 7fca7a014640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca740653b0 0x7fca7410c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:53:14.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:14.128+0000 7fca7a014640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca7410ccf0 con 0x7fca740653b0 2024-06-18T02:53:14.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:14.128+0000 7fca737fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca740653b0 0x7fca7410c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:53:14.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:14.128+0000 7fca737fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca740653b0 0x7fca7410c7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56556/0 (socket says 172.21.15.16:56556) 2024-06-18T02:53:14.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:14.128+0000 7fca737fe640 1 -- 172.21.15.16:0/2047475931 learned_addr learned my addr 172.21.15.16:0/2047475931 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:53:14.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:14.128+0000 7fca737fe640 1 -- 172.21.15.16:0/2047475931 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca64005040 con 0x7fca740653b0 2024-06-18T02:53:14.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:14.129+0000 7fca737fe640 1 --2- 172.21.15.16:0/2047475931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca740653b0 0x7fca7410c7b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fca64004820 tx=0x7fca64009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:53:14.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:14.129+0000 7fca5bfff640 1 -- 172.21.15.16:0/2047475931 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca64032650 con 0x7fca740653b0 2024-06-18T02:53:14.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:14.129+0000 7fca5bfff640 1 -- 172.21.15.16:0/2047475931 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fca64032c80 con 0x7fca740653b0 2024-06-18T02:53:14.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:14.129+0000 7fca7a014640 1 -- 172.21.15.16:0/2047475931 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca7410cef0 con 0x7fca740653b0 2024-06-18T02:53:14.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:14.129+0000 7fca5bfff640 1 -- 172.21.15.16:0/2047475931 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca64033d60 con 0x7fca740653b0 2024-06-18T02:53:14.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:14.129+0000 7fca7a014640 1 -- 172.21.15.16:0/2047475931 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca7410da80 con 0x7fca740653b0 2024-06-18T02:53:14.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:14.130+0000 7fca7a014640 1 -- 172.21.15.16:0/2047475931 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca741026b0 con 0x7fca740653b0 2024-06-18T02:53:14.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:14.130+0000 7fca5bfff640 1 -- 172.21.15.16:0/2047475931 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 42002+0+0 (secure 0 0 0) 0x7fca64030020 con 0x7fca740653b0 2024-06-18T02:53:14.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:14.131+0000 7fca5bfff640 1 --2- 172.21.15.16:0/2047475931 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7fca5c034900 0x7fca5c036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:53:14.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:14.131+0000 7fca5bfff640 1 -- 172.21.15.16:0/2047475931 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fca640667c0 con 0x7fca740653b0 2024-06-18T02:53:14.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:14.131+0000 7fca72ffd640 1 --2- 172.21.15.16:0/2047475931 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7fca5c034900 0x7fca5c036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:53:14.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:14.131+0000 7fca72ffd640 1 --2- 172.21.15.16:0/2047475931 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7fca5c034900 0x7fca5c036db0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fca68006fd0 tx=0x7fca68006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:53:14.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:14.139+0000 7fca5bfff640 1 -- 172.21.15.16:0/2047475931 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fca6403c560 con 0x7fca740653b0 2024-06-18T02:53:14.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:14.333+0000 7fca7a014640 1 -- 172.21.15.16:0/2047475931 --> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7fca740008d0 con 0x7fca5c034900 2024-06-18T02:53:14.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:14.343+0000 7fca5bfff640 1 -- 172.21.15.16:0/2047475931 <== mgr.14160 v2:172.21.15.16:6800/3749129160 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fca740008d0 con 0x7fca5c034900 2024-06-18T02:53:14.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:14.346+0000 7fca7a014640 1 -- 172.21.15.16:0/2047475931 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7fca5c034900 msgr2=0x7fca5c036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:53:14.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:14.346+0000 7fca7a014640 1 --2- 172.21.15.16:0/2047475931 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7fca5c034900 0x7fca5c036db0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fca68006fd0 tx=0x7fca68006db0 comp rx=0 tx=0).stop 2024-06-18T02:53:14.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:14.346+0000 7fca7a014640 1 -- 172.21.15.16:0/2047475931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca740653b0 msgr2=0x7fca7410c7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:53:14.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:14.346+0000 7fca7a014640 1 --2- 172.21.15.16:0/2047475931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca740653b0 0x7fca7410c7b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fca64004820 tx=0x7fca64009da0 comp rx=0 tx=0).stop 2024-06-18T02:53:14.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:14.346+0000 7fca7a014640 1 -- 172.21.15.16:0/2047475931 shutdown_connections 2024-06-18T02:53:14.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:14.346+0000 7fca7a014640 1 --2- 172.21.15.16:0/2047475931 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7fca5c034900 0x7fca5c036db0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:53:14.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:14.346+0000 7fca7a014640 1 --2- 172.21.15.16:0/2047475931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fca740653b0 0x7fca7410c7b0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:53:14.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:14.346+0000 7fca7a014640 1 -- 172.21.15.16:0/2047475931 >> 172.21.15.16:0/2047475931 conn(0x7fca740f6820 msgr2=0x7fca741032e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:53:14.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:14.346+0000 7fca7a014640 1 -- 172.21.15.16:0/2047475931 shutdown_connections 2024-06-18T02:53:14.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:14.346+0000 7fca7a014640 1 -- 172.21.15.16:0/2047475931 wait complete. 2024-06-18T02:53:15.044 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:14 smithi016 ceph-mon[29214]: from='mgr.14160 172.21.15.16:0/3988449102' entity='mgr.smithi016.amxydz' 2024-06-18T02:53:15.044 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:14 smithi016 ceph-mon[29214]: from='mgr.14160 172.21.15.16:0/3988449102' entity='mgr.smithi016.amxydz' 2024-06-18T02:53:15.044 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:14 smithi016 ceph-mon[29214]: from='mgr.14160 172.21.15.16:0/3988449102' entity='mgr.smithi016.amxydz' 2024-06-18T02:53:15.044 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:14 smithi016 ceph-mon[29214]: from='mgr.14160 172.21.15.16:0/3988449102' entity='mgr.smithi016.amxydz' 2024-06-18T02:53:15.044 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:14 smithi016 ceph-mon[29214]: Deploying daemon node-exporter.smithi016 on smithi016 2024-06-18T02:53:15.044 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:14 smithi016 ceph-mon[29214]: from='client.14184 -' entity='client.crash.smithi016' cmd=[{"prefix": "crash post", "target": ["mon-mgr", ""]}]: dispatch 2024-06-18T02:53:15.045 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:14 smithi016 ceph-mon[29214]: from='mgr.14160 172.21.15.16:0/3988449102' entity='mgr.smithi016.amxydz' 2024-06-18T02:53:15.045 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:14 smithi016 ceph-mon[29214]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-06-18T02:53:15.045 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:14 smithi016 ceph-mon[29214]: from='mgr.14160 172.21.15.16:0/3988449102' entity='mgr.smithi016.amxydz' 2024-06-18T02:53:15.045 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:14 smithi016 ceph-mon[29214]: from='mgr.14160 172.21.15.16:0/3988449102' entity='mgr.smithi016.amxydz' 2024-06-18T02:53:15.072 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi123 2024-06-18T02:53:15.072 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-06-18T02:53:15.072 DEBUG:teuthology.orchestra.run.smithi123:> dd of=/etc/ceph/ceph.conf 2024-06-18T02:53:15.092 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-06-18T02:53:15.092 DEBUG:teuthology.orchestra.run.smithi123:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-06-18T02:53:15.151 INFO:tasks.cephadm:Adding host smithi123 to orchestrator... 2024-06-18T02:53:15.151 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph orch host add smithi123 2024-06-18T02:53:15.413 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T02:53:16.305 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:15 smithi016 ceph-mon[29214]: from='client.14188 -' entity='client.crash.smithi016' cmd=[{"prefix": "crash post", "target": ["mon-mgr", ""]}]: dispatch 2024-06-18T02:53:18.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:17.959+0000 7fcd1fd64640 1 -- 172.21.15.16:0/2008829348 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd18065ea0 msgr2=0x7fcd180feba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:53:18.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:17.959+0000 7fcd1fd64640 1 --2- 172.21.15.16:0/2008829348 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd18065ea0 0x7fcd180feba0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fcd14004850 tx=0x7fcd14009da0 comp rx=0 tx=0).stop 2024-06-18T02:53:18.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:17.959+0000 7fcd1fd64640 1 -- 172.21.15.16:0/2008829348 shutdown_connections 2024-06-18T02:53:18.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:17.959+0000 7fcd1fd64640 1 --2- 172.21.15.16:0/2008829348 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd18065ea0 0x7fcd180feba0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:53:18.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:17.959+0000 7fcd1fd64640 1 -- 172.21.15.16:0/2008829348 >> 172.21.15.16:0/2008829348 conn(0x7fcd180f65b0 msgr2=0x7fcd180f8a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:53:18.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:17.959+0000 7fcd1fd64640 1 -- 172.21.15.16:0/2008829348 shutdown_connections 2024-06-18T02:53:18.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:17.959+0000 7fcd1fd64640 1 -- 172.21.15.16:0/2008829348 wait complete. 2024-06-18T02:53:18.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:17.960+0000 7fcd1fd64640 1 Processor -- start 2024-06-18T02:53:18.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:17.960+0000 7fcd1fd64640 1 -- start start 2024-06-18T02:53:18.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:17.960+0000 7fcd1fd64640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd18065ea0 0x7fcd1810c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:53:18.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:17.960+0000 7fcd1fd64640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd1810ca90 con 0x7fcd18065ea0 2024-06-18T02:53:18.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:17.960+0000 7fcd1db00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd18065ea0 0x7fcd1810c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:53:18.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:17.960+0000 7fcd1db00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd18065ea0 0x7fcd1810c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56606/0 (socket says 172.21.15.16:56606) 2024-06-18T02:53:18.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:17.960+0000 7fcd1db00640 1 -- 172.21.15.16:0/2523165613 learned_addr learned my addr 172.21.15.16:0/2523165613 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:53:18.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:17.960+0000 7fcd1db00640 1 -- 172.21.15.16:0/2523165613 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd14005040 con 0x7fcd18065ea0 2024-06-18T02:53:18.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:17.960+0000 7fcd1db00640 1 --2- 172.21.15.16:0/2523165613 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd18065ea0 0x7fcd1810c550 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fcd14004820 tx=0x7fcd14009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:53:18.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:17.960+0000 7fcd0a7fc640 1 -- 172.21.15.16:0/2523165613 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd14032650 con 0x7fcd18065ea0 2024-06-18T02:53:18.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:17.961+0000 7fcd0a7fc640 1 -- 172.21.15.16:0/2523165613 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcd14032c80 con 0x7fcd18065ea0 2024-06-18T02:53:18.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:17.961+0000 7fcd1fd64640 1 -- 172.21.15.16:0/2523165613 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd1810cc90 con 0x7fcd18065ea0 2024-06-18T02:53:18.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:17.961+0000 7fcd0a7fc640 1 -- 172.21.15.16:0/2523165613 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd14033d60 con 0x7fcd18065ea0 2024-06-18T02:53:18.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:17.961+0000 7fcd1fd64640 1 -- 172.21.15.16:0/2523165613 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd1810d820 con 0x7fcd18065ea0 2024-06-18T02:53:18.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:17.961+0000 7fcd0a7fc640 1 -- 172.21.15.16:0/2523165613 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 42002+0+0 (secure 0 0 0) 0x7fcd14030020 con 0x7fcd18065ea0 2024-06-18T02:53:18.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:17.962+0000 7fcd0a7fc640 1 --2- 172.21.15.16:0/2523165613 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7fcd04034900 0x7fcd04036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:53:18.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:17.962+0000 7fcd0a7fc640 1 -- 172.21.15.16:0/2523165613 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fcd14066720 con 0x7fcd18065ea0 2024-06-18T02:53:18.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:17.962+0000 7fcd1fd64640 1 -- 172.21.15.16:0/2523165613 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd18102490 con 0x7fcd18065ea0 2024-06-18T02:53:18.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:17.962+0000 7fcd1d2ff640 1 --2- 172.21.15.16:0/2523165613 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7fcd04034900 0x7fcd04036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:53:18.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:17.962+0000 7fcd1d2ff640 1 --2- 172.21.15.16:0/2523165613 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7fcd04034900 0x7fcd04036db0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fcd0c006fd0 tx=0x7fcd0c006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:53:18.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:17.966+0000 7fcd0a7fc640 1 -- 172.21.15.16:0/2523165613 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fcd1403c380 con 0x7fcd18065ea0 2024-06-18T02:53:18.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:18.144+0000 7fcd1fd64640 1 -- 172.21.15.16:0/2523165613 --> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi123", "target": ["mon-mgr", ""]}) v1 -- 0x7fcd18101e70 con 0x7fcd04034900 2024-06-18T02:53:19.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:19 smithi016 ceph-mon[29214]: from='client.14190 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi123", "target": ["mon-mgr", ""]}]: dispatch 2024-06-18T02:53:19.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:19 smithi016 ceph-mon[29214]: from='mgr.14160 172.21.15.16:0/3988449102' entity='mgr.smithi016.amxydz' 2024-06-18T02:53:19.556 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:19 smithi016 ceph-mon[29214]: from='mgr.14160 172.21.15.16:0/3988449102' entity='mgr.smithi016.amxydz' 2024-06-18T02:53:19.556 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:19 smithi016 ceph-mon[29214]: from='mgr.14160 172.21.15.16:0/3988449102' entity='mgr.smithi016.amxydz' 2024-06-18T02:53:19.556 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:19 smithi016 ceph-mon[29214]: from='mgr.14160 172.21.15.16:0/3988449102' entity='mgr.smithi016.amxydz' 2024-06-18T02:53:19.556 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:19 smithi016 ceph-mon[29214]: from='mgr.14160 172.21.15.16:0/3988449102' entity='mgr.smithi016.amxydz' 2024-06-18T02:53:20.305 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:20 smithi016 ceph-mon[29214]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:53:20.305 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:20 smithi016 ceph-mon[29214]: Deploying daemon alertmanager.smithi016 on smithi016 2024-06-18T02:53:20.305 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:20 smithi016 ceph-mon[29214]: Health check failed: 2 mgr modules have recently crashed (RECENT_MGR_MODULE_CRASH) 2024-06-18T02:53:20.305 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:20 smithi016 ceph-mon[29214]: Deploying cephadm binary to smithi123 2024-06-18T02:53:21.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:21.646+0000 7fcd0a7fc640 1 -- 172.21.15.16:0/2523165613 <== mgr.14160 v2:172.21.15.16:6800/3749129160 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fcd0c009510 con 0x7fcd04034900 2024-06-18T02:53:21.647 INFO:teuthology.orchestra.run.smithi016.stdout:Added host 'smithi123' with addr '172.21.15.123' 2024-06-18T02:53:21.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:21.648+0000 7fcd1fd64640 1 -- 172.21.15.16:0/2523165613 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7fcd04034900 msgr2=0x7fcd04036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:53:21.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:21.648+0000 7fcd1fd64640 1 --2- 172.21.15.16:0/2523165613 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7fcd04034900 0x7fcd04036db0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fcd0c006fd0 tx=0x7fcd0c006db0 comp rx=0 tx=0).stop 2024-06-18T02:53:21.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:21.648+0000 7fcd1fd64640 1 -- 172.21.15.16:0/2523165613 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd18065ea0 msgr2=0x7fcd1810c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:53:21.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:21.648+0000 7fcd1fd64640 1 --2- 172.21.15.16:0/2523165613 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd18065ea0 0x7fcd1810c550 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fcd14004820 tx=0x7fcd14009da0 comp rx=0 tx=0).stop 2024-06-18T02:53:21.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:21.648+0000 7fcd1fd64640 1 -- 172.21.15.16:0/2523165613 shutdown_connections 2024-06-18T02:53:21.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:21.648+0000 7fcd1fd64640 1 --2- 172.21.15.16:0/2523165613 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7fcd04034900 0x7fcd04036db0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:53:21.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:21.648+0000 7fcd1fd64640 1 --2- 172.21.15.16:0/2523165613 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd18065ea0 0x7fcd1810c550 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:53:21.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:21.648+0000 7fcd1fd64640 1 -- 172.21.15.16:0/2523165613 >> 172.21.15.16:0/2523165613 conn(0x7fcd180f65b0 msgr2=0x7fcd181030c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:53:21.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:21.648+0000 7fcd1fd64640 1 -- 172.21.15.16:0/2523165613 shutdown_connections 2024-06-18T02:53:21.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:21.648+0000 7fcd1fd64640 1 -- 172.21.15.16:0/2523165613 wait complete. 2024-06-18T02:53:22.261 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:22 smithi016 ceph-mon[29214]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:53:22.261 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:22 smithi016 ceph-mon[29214]: from='mgr.14160 172.21.15.16:0/3988449102' entity='mgr.smithi016.amxydz' 2024-06-18T02:53:22.290 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph orch host ls --format=json 2024-06-18T02:53:22.552 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T02:53:23.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:23 smithi016 ceph-mon[29214]: Added host smithi123 2024-06-18T02:53:24.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:24.193+0000 7f171e3b4640 1 -- 172.21.15.16:0/1854053915 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17180712d0 msgr2=0x7f171806f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:53:24.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:24.193+0000 7f171e3b4640 1 --2- 172.21.15.16:0/1854053915 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17180712d0 0x7f171806f800 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1708004850 tx=0x7f1708009da0 comp rx=0 tx=0).stop 2024-06-18T02:53:24.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:24.193+0000 7f171e3b4640 1 -- 172.21.15.16:0/1854053915 shutdown_connections 2024-06-18T02:53:24.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:24.193+0000 7f171e3b4640 1 --2- 172.21.15.16:0/1854053915 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17180712d0 0x7f171806f800 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:53:24.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:24.193+0000 7f171e3b4640 1 -- 172.21.15.16:0/1854053915 >> 172.21.15.16:0/1854053915 conn(0x7f17180f76f0 msgr2=0x7f17180f9b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:53:24.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:24.193+0000 7f171e3b4640 1 -- 172.21.15.16:0/1854053915 shutdown_connections 2024-06-18T02:53:24.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:24.193+0000 7f171e3b4640 1 -- 172.21.15.16:0/1854053915 wait complete. 2024-06-18T02:53:24.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:24.194+0000 7f171e3b4640 1 Processor -- start 2024-06-18T02:53:24.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:24.194+0000 7f171e3b4640 1 -- start start 2024-06-18T02:53:24.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:24.194+0000 7f171e3b4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17180712d0 0x7f171810e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:53:24.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:24.194+0000 7f171e3b4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f171810ed60 con 0x7f17180712d0 2024-06-18T02:53:24.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:24.194+0000 7f1717fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17180712d0 0x7f171810e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:53:24.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:24.194+0000 7f1717fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17180712d0 0x7f171810e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54316/0 (socket says 172.21.15.16:54316) 2024-06-18T02:53:24.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:24.194+0000 7f1717fff640 1 -- 172.21.15.16:0/498352017 learned_addr learned my addr 172.21.15.16:0/498352017 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:53:24.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:24.194+0000 7f1717fff640 1 -- 172.21.15.16:0/498352017 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1708005040 con 0x7f17180712d0 2024-06-18T02:53:24.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:24.195+0000 7f1717fff640 1 --2- 172.21.15.16:0/498352017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17180712d0 0x7f171810e820 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1708004820 tx=0x7f1708009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:53:24.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:24.195+0000 7f1714ff9640 1 -- 172.21.15.16:0/498352017 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1708032650 con 0x7f17180712d0 2024-06-18T02:53:24.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:24.195+0000 7f171e3b4640 1 -- 172.21.15.16:0/498352017 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f171810ef60 con 0x7f17180712d0 2024-06-18T02:53:24.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:24.195+0000 7f1714ff9640 1 -- 172.21.15.16:0/498352017 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1708032c80 con 0x7f17180712d0 2024-06-18T02:53:24.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:24.195+0000 7f1714ff9640 1 -- 172.21.15.16:0/498352017 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1708033d60 con 0x7f17180712d0 2024-06-18T02:53:24.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:24.195+0000 7f171e3b4640 1 -- 172.21.15.16:0/498352017 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f171810faf0 con 0x7f17180712d0 2024-06-18T02:53:24.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:24.196+0000 7f1714ff9640 1 -- 172.21.15.16:0/498352017 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 42002+0+0 (secure 0 0 0) 0x7f1708030020 con 0x7f17180712d0 2024-06-18T02:53:24.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:24.196+0000 7f171e3b4640 1 -- 172.21.15.16:0/498352017 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1718104680 con 0x7f17180712d0 2024-06-18T02:53:24.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:24.196+0000 7f1714ff9640 1 --2- 172.21.15.16:0/498352017 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f1700034900 0x7f1700036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:53:24.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:24.196+0000 7f1714ff9640 1 -- 172.21.15.16:0/498352017 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f1708066810 con 0x7f17180712d0 2024-06-18T02:53:24.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:24.196+0000 7f17177fe640 1 --2- 172.21.15.16:0/498352017 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f1700034900 0x7f1700036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:53:24.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:24.197+0000 7f17177fe640 1 --2- 172.21.15.16:0/498352017 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f1700034900 0x7f1700036db0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f170c006fd0 tx=0x7f170c006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:53:24.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:24.201+0000 7f1714ff9640 1 -- 172.21.15.16:0/498352017 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f170803c380 con 0x7f17180712d0 2024-06-18T02:53:24.305 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:24 smithi016 ceph-mon[29214]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:53:24.305 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:24 smithi016 ceph-mon[29214]: from='mgr.14160 172.21.15.16:0/3988449102' entity='mgr.smithi016.amxydz' 2024-06-18T02:53:24.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:24.382+0000 7f171e3b4640 1 -- 172.21.15.16:0/498352017 --> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f171810fd30 con 0x7f1700034900 2024-06-18T02:53:24.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:24.383+0000 7f1714ff9640 1 -- 172.21.15.16:0/498352017 <== mgr.14160 v2:172.21.15.16:6800/3749129160 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+160 (secure 0 0 0) 0x7f171810fd30 con 0x7f1700034900 2024-06-18T02:53:24.384 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:53:24.384 INFO:teuthology.orchestra.run.smithi016.stdout:[{"addr": "172.21.15.16", "hostname": "smithi016", "labels": [], "status": ""}, {"addr": "172.21.15.123", "hostname": "smithi123", "labels": [], "status": ""}] 2024-06-18T02:53:24.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:24.385+0000 7f171e3b4640 1 -- 172.21.15.16:0/498352017 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f1700034900 msgr2=0x7f1700036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:53:24.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:24.385+0000 7f171e3b4640 1 --2- 172.21.15.16:0/498352017 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f1700034900 0x7f1700036db0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f170c006fd0 tx=0x7f170c006db0 comp rx=0 tx=0).stop 2024-06-18T02:53:24.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:24.385+0000 7f171e3b4640 1 -- 172.21.15.16:0/498352017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17180712d0 msgr2=0x7f171810e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:53:24.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:24.385+0000 7f171e3b4640 1 --2- 172.21.15.16:0/498352017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17180712d0 0x7f171810e820 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1708004820 tx=0x7f1708009da0 comp rx=0 tx=0).stop 2024-06-18T02:53:24.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:24.385+0000 7f171e3b4640 1 -- 172.21.15.16:0/498352017 shutdown_connections 2024-06-18T02:53:24.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:24.385+0000 7f171e3b4640 1 --2- 172.21.15.16:0/498352017 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f1700034900 0x7f1700036db0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:53:24.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:24.385+0000 7f171e3b4640 1 --2- 172.21.15.16:0/498352017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17180712d0 0x7f171810e820 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:53:24.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:24.385+0000 7f171e3b4640 1 -- 172.21.15.16:0/498352017 >> 172.21.15.16:0/498352017 conn(0x7f17180f76f0 msgr2=0x7f17181052b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:53:24.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:24.386+0000 7f171e3b4640 1 -- 172.21.15.16:0/498352017 shutdown_connections 2024-06-18T02:53:24.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:24.386+0000 7f171e3b4640 1 -- 172.21.15.16:0/498352017 wait complete. 2024-06-18T02:53:25.119 INFO:tasks.cephadm:Setting crush tunables to default 2024-06-18T02:53:25.119 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph osd crush tunables default 2024-06-18T02:53:25.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:25 smithi016 ceph-mon[29214]: from='client.14192 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-18T02:53:26.091 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T02:53:26.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:26 smithi016 ceph-mon[29214]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:53:28.305 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:28 smithi016 ceph-mon[29214]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:53:29.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:29.395+0000 7f203de4a640 1 -- 172.21.15.16:0/3388976040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2038065330 msgr2=0x7f20380fedd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:53:29.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:29.395+0000 7f203de4a640 1 --2- 172.21.15.16:0/3388976040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2038065330 0x7f20380fedd0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2028004850 tx=0x7f2028009da0 comp rx=0 tx=0).stop 2024-06-18T02:53:29.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:29.395+0000 7f203de4a640 1 -- 172.21.15.16:0/3388976040 shutdown_connections 2024-06-18T02:53:29.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:29.395+0000 7f203de4a640 1 --2- 172.21.15.16:0/3388976040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2038065330 0x7f20380fedd0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:53:29.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:29.395+0000 7f203de4a640 1 -- 172.21.15.16:0/3388976040 >> 172.21.15.16:0/3388976040 conn(0x7f20380f6820 msgr2=0x7f20380f8c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:53:29.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:29.395+0000 7f203de4a640 1 -- 172.21.15.16:0/3388976040 shutdown_connections 2024-06-18T02:53:29.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:29.395+0000 7f203de4a640 1 -- 172.21.15.16:0/3388976040 wait complete. 2024-06-18T02:53:29.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:29.395+0000 7f203de4a640 1 Processor -- start 2024-06-18T02:53:29.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:29.396+0000 7f203de4a640 1 -- start start 2024-06-18T02:53:29.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:29.396+0000 7f203de4a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2038065330 0x7f203810c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:53:29.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:29.396+0000 7f203de4a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f203810cce0 con 0x7f2038065330 2024-06-18T02:53:29.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:29.396+0000 7f20377fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2038065330 0x7f203810c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:53:29.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:29.396+0000 7f20377fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2038065330 0x7f203810c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54334/0 (socket says 172.21.15.16:54334) 2024-06-18T02:53:29.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:29.396+0000 7f20377fe640 1 -- 172.21.15.16:0/2371331266 learned_addr learned my addr 172.21.15.16:0/2371331266 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:53:29.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:29.396+0000 7f20377fe640 1 -- 172.21.15.16:0/2371331266 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2028005040 con 0x7f2038065330 2024-06-18T02:53:29.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:29.396+0000 7f20377fe640 1 --2- 172.21.15.16:0/2371331266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2038065330 0x7f203810c7a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f2028004820 tx=0x7f2028009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:53:29.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:29.396+0000 7f201ffff640 1 -- 172.21.15.16:0/2371331266 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2028032650 con 0x7f2038065330 2024-06-18T02:53:29.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:29.397+0000 7f203de4a640 1 -- 172.21.15.16:0/2371331266 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f203810cee0 con 0x7f2038065330 2024-06-18T02:53:29.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:29.397+0000 7f201ffff640 1 -- 172.21.15.16:0/2371331266 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2028032c80 con 0x7f2038065330 2024-06-18T02:53:29.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:29.397+0000 7f201ffff640 1 -- 172.21.15.16:0/2371331266 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2028033d60 con 0x7f2038065330 2024-06-18T02:53:29.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:29.397+0000 7f203de4a640 1 -- 172.21.15.16:0/2371331266 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f203810da70 con 0x7f2038065330 2024-06-18T02:53:29.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:29.397+0000 7f201ffff640 1 -- 172.21.15.16:0/2371331266 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 42002+0+0 (secure 0 0 0) 0x7f2028030020 con 0x7f2038065330 2024-06-18T02:53:29.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:29.398+0000 7f201ffff640 1 --2- 172.21.15.16:0/2371331266 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f2020034900 0x7f2020036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:53:29.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:29.398+0000 7f203de4a640 1 -- 172.21.15.16:0/2371331266 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f203810d3b0 con 0x7f2038065330 2024-06-18T02:53:29.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:29.398+0000 7f201ffff640 1 -- 172.21.15.16:0/2371331266 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f20280667c0 con 0x7f2038065330 2024-06-18T02:53:29.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:29.398+0000 7f2036ffd640 1 --2- 172.21.15.16:0/2371331266 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f2020034900 0x7f2020036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:53:29.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:29.398+0000 7f2036ffd640 1 --2- 172.21.15.16:0/2371331266 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f2020034900 0x7f2020036db0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f202c006fd0 tx=0x7f202c006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:53:29.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:29.403+0000 7f201ffff640 1 -- 172.21.15.16:0/2371331266 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f203810d3b0 con 0x7f2038065330 2024-06-18T02:53:29.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:29.583+0000 7f203de4a640 1 -- 172.21.15.16:0/2371331266 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f203810d3b0 con 0x7f2038065330 2024-06-18T02:53:30.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:30.217+0000 7f201ffff640 1 -- 172.21.15.16:0/2371331266 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f203810d3b0 con 0x7f2038065330 2024-06-18T02:53:30.218 INFO:teuthology.orchestra.run.smithi016.stderr:adjusted tunables profile to default 2024-06-18T02:53:30.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:30.220+0000 7f203de4a640 1 -- 172.21.15.16:0/2371331266 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f2020034900 msgr2=0x7f2020036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:53:30.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:30.220+0000 7f203de4a640 1 --2- 172.21.15.16:0/2371331266 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f2020034900 0x7f2020036db0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f202c006fd0 tx=0x7f202c006db0 comp rx=0 tx=0).stop 2024-06-18T02:53:30.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:30.220+0000 7f203de4a640 1 -- 172.21.15.16:0/2371331266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2038065330 msgr2=0x7f203810c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:53:30.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:30.220+0000 7f203de4a640 1 --2- 172.21.15.16:0/2371331266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2038065330 0x7f203810c7a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f2028004820 tx=0x7f2028009da0 comp rx=0 tx=0).stop 2024-06-18T02:53:30.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:30.220+0000 7f203de4a640 1 -- 172.21.15.16:0/2371331266 shutdown_connections 2024-06-18T02:53:30.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:30.220+0000 7f203de4a640 1 --2- 172.21.15.16:0/2371331266 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f2020034900 0x7f2020036db0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:53:30.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:30.220+0000 7f203de4a640 1 --2- 172.21.15.16:0/2371331266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2038065330 0x7f203810c7a0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:53:30.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:30.220+0000 7f203de4a640 1 -- 172.21.15.16:0/2371331266 >> 172.21.15.16:0/2371331266 conn(0x7f20380f6820 msgr2=0x7f20381032f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:53:30.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:30.221+0000 7f203de4a640 1 -- 172.21.15.16:0/2371331266 shutdown_connections 2024-06-18T02:53:30.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:53:30.221+0000 7f203de4a640 1 -- 172.21.15.16:0/2371331266 wait complete. 2024-06-18T02:53:30.482 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:30 smithi016 ceph-mon[29214]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:53:30.483 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:30 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/2371331266' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-06-18T02:53:31.187 INFO:tasks.cephadm:Adding mon.smithi016 on smithi016 2024-06-18T02:53:31.187 INFO:tasks.cephadm:Adding mon.smithi123 on smithi123 2024-06-18T02:53:31.187 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph orch apply mon '2;smithi016:172.21.15.16=smithi016;smithi123:172.21.15.123=smithi123' 2024-06-18T02:53:31.429 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:31 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/2371331266' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-06-18T02:53:31.429 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:31 smithi016 ceph-mon[29214]: osdmap e4: 0 total, 0 up, 0 in 2024-06-18T02:53:31.433 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-18T02:53:31.480 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-18T02:53:32.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:32 smithi016 ceph-mon[29214]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:53:32.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:32 smithi016 ceph-mon[29214]: from='mgr.14160 172.21.15.16:0/3988449102' entity='mgr.smithi016.amxydz' 2024-06-18T02:53:32.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:32 smithi016 ceph-mon[29214]: from='mgr.14160 172.21.15.16:0/3988449102' entity='mgr.smithi016.amxydz' 2024-06-18T02:53:32.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:32 smithi016 ceph-mon[29214]: from='mgr.14160 172.21.15.16:0/3988449102' entity='mgr.smithi016.amxydz' 2024-06-18T02:53:32.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:32 smithi016 ceph-mon[29214]: from='mgr.14160 172.21.15.16:0/3988449102' entity='mgr.smithi016.amxydz' 2024-06-18T02:53:32.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:32 smithi016 ceph-mon[29214]: from='mgr.14160 172.21.15.16:0/3988449102' entity='mgr.smithi016.amxydz' 2024-06-18T02:53:32.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:32 smithi016 ceph-mon[29214]: from='mgr.14160 172.21.15.16:0/3988449102' entity='mgr.smithi016.amxydz' 2024-06-18T02:53:32.556 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:32 smithi016 ceph-mon[29214]: from='mgr.14160 172.21.15.16:0/3988449102' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-06-18T02:53:32.556 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:32 smithi016 ceph-mon[29214]: from='mgr.14160 172.21.15.16:0/3988449102' entity='mgr.smithi016.amxydz' 2024-06-18T02:53:32.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:32.820+0000 7efefc17d640 1 -- 172.21.15.123:0/221815222 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efef4065ea0 msgr2=0x7efef40febb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:53:32.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:32.820+0000 7efefc17d640 1 --2- 172.21.15.123:0/221815222 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efef4065ea0 0x7efef40febb0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7efef0004850 tx=0x7efef0009da0 comp rx=0 tx=0).stop 2024-06-18T02:53:32.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:32.820+0000 7efefc17d640 1 -- 172.21.15.123:0/221815222 shutdown_connections 2024-06-18T02:53:32.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:32.820+0000 7efefc17d640 1 --2- 172.21.15.123:0/221815222 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efef4065ea0 0x7efef40febb0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:53:32.831 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:32.820+0000 7efefc17d640 1 -- 172.21.15.123:0/221815222 >> 172.21.15.123:0/221815222 conn(0x7efef40f65e0 msgr2=0x7efef40f8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:53:32.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:32.820+0000 7efefc17d640 1 -- 172.21.15.123:0/221815222 shutdown_connections 2024-06-18T02:53:32.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:32.820+0000 7efefc17d640 1 -- 172.21.15.123:0/221815222 wait complete. 2024-06-18T02:53:32.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:32.821+0000 7efefc17d640 1 Processor -- start 2024-06-18T02:53:32.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:32.821+0000 7efefc17d640 1 -- start start 2024-06-18T02:53:32.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:32.821+0000 7efefc17d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efef4065ea0 0x7efef40fc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:53:32.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:32.821+0000 7efefc17d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef40fca20 con 0x7efef4065ea0 2024-06-18T02:53:32.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:32.821+0000 7efef9f19640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efef4065ea0 0x7efef40fc4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:53:32.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:32.821+0000 7efef9f19640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efef4065ea0 0x7efef40fc4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.123:48342/0 (socket says 172.21.15.123:48342) 2024-06-18T02:53:32.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:32.821+0000 7efef9f19640 1 -- 172.21.15.123:0/1141317781 learned_addr learned my addr 172.21.15.123:0/1141317781 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-06-18T02:53:32.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:32.821+0000 7efef9f19640 1 -- 172.21.15.123:0/1141317781 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efef0005040 con 0x7efef4065ea0 2024-06-18T02:53:32.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:32.822+0000 7efef9f19640 1 --2- 172.21.15.123:0/1141317781 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efef4065ea0 0x7efef40fc4e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7efef00098d0 tx=0x7efef0009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:53:32.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:32.822+0000 7efee67fc640 1 -- 172.21.15.123:0/1141317781 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efef0032650 con 0x7efef4065ea0 2024-06-18T02:53:32.832 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:32.822+0000 7efee67fc640 1 -- 172.21.15.123:0/1141317781 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efef0032c80 con 0x7efef4065ea0 2024-06-18T02:53:32.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:32.822+0000 7efefc17d640 1 -- 172.21.15.123:0/1141317781 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efef40faab0 con 0x7efef4065ea0 2024-06-18T02:53:32.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:32.822+0000 7efee67fc640 1 -- 172.21.15.123:0/1141317781 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efef003dc50 con 0x7efef4065ea0 2024-06-18T02:53:32.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:32.822+0000 7efefc17d640 1 -- 172.21.15.123:0/1141317781 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efef40faf20 con 0x7efef4065ea0 2024-06-18T02:53:32.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:32.822+0000 7efee67fc640 1 -- 172.21.15.123:0/1141317781 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 42002+0+0 (secure 0 0 0) 0x7efef0030020 con 0x7efef4065ea0 2024-06-18T02:53:32.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:32.823+0000 7efefc17d640 1 -- 172.21.15.123:0/1141317781 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efef41024a0 con 0x7efef4065ea0 2024-06-18T02:53:32.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:32.823+0000 7efee67fc640 1 --2- 172.21.15.123:0/1141317781 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7efee0034950 0x7efee0036e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:53:32.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:32.823+0000 7efee67fc640 1 -- 172.21.15.123:0/1141317781 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7efef0066f90 con 0x7efef4065ea0 2024-06-18T02:53:32.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:32.823+0000 7efef9718640 1 --2- 172.21.15.123:0/1141317781 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7efee0034950 0x7efee0036e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:53:32.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:32.823+0000 7efef9718640 1 --2- 172.21.15.123:0/1141317781 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7efee0034950 0x7efee0036e00 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7efee8006fd0 tx=0x7efee8006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:53:32.833 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:32.828+0000 7efee67fc640 1 -- 172.21.15.123:0/1141317781 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7efef0045570 con 0x7efef4065ea0 2024-06-18T02:53:33.001 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:33.001+0000 7efefc17d640 1 -- 172.21.15.123:0/1141317781 --> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi016:172.21.15.16=smithi016;smithi123:172.21.15.123=smithi123", "target": ["mon-mgr", ""]}) v1 -- 0x7efef40fb160 con 0x7efee0034950 2024-06-18T02:53:33.006 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:33.006+0000 7efee67fc640 1 -- 172.21.15.123:0/1141317781 <== mgr.14160 v2:172.21.15.16:6800/3749129160 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7efef40fb160 con 0x7efee0034950 2024-06-18T02:53:33.006 INFO:teuthology.orchestra.run.smithi123.stdout:Scheduled mon update... 2024-06-18T02:53:33.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:33.008+0000 7efefc17d640 1 -- 172.21.15.123:0/1141317781 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7efee0034950 msgr2=0x7efee0036e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:53:33.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:33.008+0000 7efefc17d640 1 --2- 172.21.15.123:0/1141317781 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7efee0034950 0x7efee0036e00 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7efee8006fd0 tx=0x7efee8006db0 comp rx=0 tx=0).stop 2024-06-18T02:53:33.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:33.008+0000 7efefc17d640 1 -- 172.21.15.123:0/1141317781 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efef4065ea0 msgr2=0x7efef40fc4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:53:33.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:33.008+0000 7efefc17d640 1 --2- 172.21.15.123:0/1141317781 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efef4065ea0 0x7efef40fc4e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7efef00098d0 tx=0x7efef0009da0 comp rx=0 tx=0).stop 2024-06-18T02:53:33.008 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:33.008+0000 7efefc17d640 1 -- 172.21.15.123:0/1141317781 shutdown_connections 2024-06-18T02:53:33.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:33.008+0000 7efefc17d640 1 --2- 172.21.15.123:0/1141317781 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7efee0034950 0x7efee0036e00 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:53:33.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:33.008+0000 7efefc17d640 1 --2- 172.21.15.123:0/1141317781 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efef4065ea0 0x7efef40fc4e0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:53:33.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:33.008+0000 7efefc17d640 1 -- 172.21.15.123:0/1141317781 >> 172.21.15.123:0/1141317781 conn(0x7efef40f65e0 msgr2=0x7efef41030d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:53:33.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:33.008+0000 7efefc17d640 1 -- 172.21.15.123:0/1141317781 shutdown_connections 2024-06-18T02:53:33.009 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:33.008+0000 7efefc17d640 1 -- 172.21.15.123:0/1141317781 wait complete. 2024-06-18T02:53:33.549 DEBUG:teuthology.orchestra.run.smithi123:mon.smithi123> sudo journalctl -f -n 0 -u ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968@mon.smithi123.service 2024-06-18T02:53:33.552 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-18T02:53:33.552 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph mon dump -f json 2024-06-18T02:53:33.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:33 smithi016 ceph-mon[29214]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-06-18T02:53:33.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:33 smithi016 ceph-mon[29214]: Deploying daemon grafana.smithi016 on smithi016 2024-06-18T02:53:33.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:33 smithi016 ceph-mon[29214]: from='mgr.14160 172.21.15.16:0/3988449102' entity='mgr.smithi016.amxydz' 2024-06-18T02:53:33.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:33 smithi016 ceph-mon[29214]: from='mgr.14160 172.21.15.16:0/3988449102' entity='mgr.smithi016.amxydz' 2024-06-18T02:53:33.826 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-18T02:53:33.872 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-18T02:53:34.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:34 smithi016 ceph-mon[29214]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:53:34.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:34 smithi016 ceph-mon[29214]: from='client.14196 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi016:172.21.15.16=smithi016;smithi123:172.21.15.123=smithi123", "target": ["mon-mgr", ""]}]: dispatch 2024-06-18T02:53:34.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:34 smithi016 ceph-mon[29214]: Saving service mon spec with placement smithi016:172.21.15.16=smithi016;smithi123:172.21.15.123=smithi123;count:2 2024-06-18T02:53:35.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:35.194+0000 7f0c8c3a3640 1 -- 172.21.15.123:0/1180018926 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c840fa5c0 msgr2=0x7f0c840fa9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:53:35.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:35.194+0000 7f0c8c3a3640 1 --2- 172.21.15.123:0/1180018926 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c840fa5c0 0x7f0c840fa9d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0c74004850 tx=0x7f0c74009da0 comp rx=0 tx=0).stop 2024-06-18T02:53:35.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:35.194+0000 7f0c8c3a3640 1 -- 172.21.15.123:0/1180018926 shutdown_connections 2024-06-18T02:53:35.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:35.194+0000 7f0c8c3a3640 1 --2- 172.21.15.123:0/1180018926 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c840fa5c0 0x7f0c840fa9d0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:53:35.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:35.194+0000 7f0c8c3a3640 1 -- 172.21.15.123:0/1180018926 >> 172.21.15.123:0/1180018926 conn(0x7f0c840f5da0 msgr2=0x7f0c840f81b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:53:35.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:35.194+0000 7f0c8c3a3640 1 -- 172.21.15.123:0/1180018926 shutdown_connections 2024-06-18T02:53:35.195 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:35.194+0000 7f0c8c3a3640 1 -- 172.21.15.123:0/1180018926 wait complete. 2024-06-18T02:53:35.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:35.195+0000 7f0c8c3a3640 1 Processor -- start 2024-06-18T02:53:35.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:35.195+0000 7f0c8c3a3640 1 -- start start 2024-06-18T02:53:35.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:35.195+0000 7f0c8c3a3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c840fa5c0 0x7f0c840f6d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:53:35.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:35.195+0000 7f0c8c3a3640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c8410f1f0 con 0x7f0c840fa5c0 2024-06-18T02:53:35.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:35.195+0000 7f0c8a13f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c840fa5c0 0x7f0c840f6d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:53:35.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:35.195+0000 7f0c8a13f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c840fa5c0 0x7f0c840f6d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.123:48364/0 (socket says 172.21.15.123:48364) 2024-06-18T02:53:35.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:35.195+0000 7f0c8a13f640 1 -- 172.21.15.123:0/2717117496 learned_addr learned my addr 172.21.15.123:0/2717117496 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-06-18T02:53:35.196 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:35.195+0000 7f0c8a13f640 1 -- 172.21.15.123:0/2717117496 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c74005040 con 0x7f0c840fa5c0 2024-06-18T02:53:35.197 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:35.195+0000 7f0c8a13f640 1 --2- 172.21.15.123:0/2717117496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c840fa5c0 0x7f0c840f6d10 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0c74004820 tx=0x7f0c74009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:53:35.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:35.196+0000 7f0c7affd640 1 -- 172.21.15.123:0/2717117496 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c74032650 con 0x7f0c840fa5c0 2024-06-18T02:53:35.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:35.196+0000 7f0c7affd640 1 -- 172.21.15.123:0/2717117496 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0c74032c80 con 0x7f0c840fa5c0 2024-06-18T02:53:35.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:35.196+0000 7f0c8c3a3640 1 -- 172.21.15.123:0/2717117496 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c8410f330 con 0x7f0c840fa5c0 2024-06-18T02:53:35.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:35.196+0000 7f0c7affd640 1 -- 172.21.15.123:0/2717117496 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c7403eda0 con 0x7f0c840fa5c0 2024-06-18T02:53:35.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:35.196+0000 7f0c8c3a3640 1 -- 172.21.15.123:0/2717117496 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c8410f800 con 0x7f0c840fa5c0 2024-06-18T02:53:35.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:35.196+0000 7f0c7affd640 1 -- 172.21.15.123:0/2717117496 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 42002+0+0 (secure 0 0 0) 0x7f0c74030020 con 0x7f0c840fa5c0 2024-06-18T02:53:35.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:35.197+0000 7f0c7affd640 1 --2- 172.21.15.123:0/2717117496 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f0c70034900 0x7f0c70036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:53:35.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:35.197+0000 7f0c7affd640 1 -- 172.21.15.123:0/2717117496 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f0c740669c0 con 0x7f0c840fa5c0 2024-06-18T02:53:35.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:35.197+0000 7f0c8c3a3640 1 -- 172.21.15.123:0/2717117496 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c8405ead0 con 0x7f0c840fa5c0 2024-06-18T02:53:35.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:35.197+0000 7f0c8993e640 1 --2- 172.21.15.123:0/2717117496 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f0c70034900 0x7f0c70036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:53:35.198 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:35.197+0000 7f0c8993e640 1 --2- 172.21.15.123:0/2717117496 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f0c70034900 0x7f0c70036db0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0c80006fd0 tx=0x7f0c80006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:53:35.201 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:35.202+0000 7f0c7affd640 1 -- 172.21.15.123:0/2717117496 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f0c74033420 con 0x7f0c840fa5c0 2024-06-18T02:53:35.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:35.427+0000 7f0c8c3a3640 1 -- 172.21.15.123:0/2717117496 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0c84104e40 con 0x7f0c840fa5c0 2024-06-18T02:53:35.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:35.428+0000 7f0c7affd640 1 -- 172.21.15.123:0/2717117496 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f0c7403c330 con 0x7f0c840fa5c0 2024-06-18T02:53:35.428 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:53:35.428 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"9e93709a-2d1d-11ef-bc9d-c7b262605968","modified":"2024-06-18T02:51:51.645370Z","created":"2024-06-18T02:51:51.645370Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi016","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-18T02:53:35.428 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-06-18T02:53:35.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:35.429+0000 7f0c8c3a3640 1 -- 172.21.15.123:0/2717117496 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f0c70034900 msgr2=0x7f0c70036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:53:35.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:35.429+0000 7f0c8c3a3640 1 --2- 172.21.15.123:0/2717117496 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f0c70034900 0x7f0c70036db0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0c80006fd0 tx=0x7f0c80006db0 comp rx=0 tx=0).stop 2024-06-18T02:53:35.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:35.429+0000 7f0c8c3a3640 1 -- 172.21.15.123:0/2717117496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c840fa5c0 msgr2=0x7f0c840f6d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:53:35.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:35.429+0000 7f0c8c3a3640 1 --2- 172.21.15.123:0/2717117496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c840fa5c0 0x7f0c840f6d10 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0c74004820 tx=0x7f0c74009da0 comp rx=0 tx=0).stop 2024-06-18T02:53:35.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:35.429+0000 7f0c8c3a3640 1 -- 172.21.15.123:0/2717117496 shutdown_connections 2024-06-18T02:53:35.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:35.429+0000 7f0c8c3a3640 1 --2- 172.21.15.123:0/2717117496 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f0c70034900 0x7f0c70036db0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:53:35.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:35.429+0000 7f0c8c3a3640 1 --2- 172.21.15.123:0/2717117496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c840fa5c0 0x7f0c840f6d10 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:53:35.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:35.429+0000 7f0c8c3a3640 1 -- 172.21.15.123:0/2717117496 >> 172.21.15.123:0/2717117496 conn(0x7f0c840f5da0 msgr2=0x7f0c840fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:53:35.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:35.430+0000 7f0c8c3a3640 1 -- 172.21.15.123:0/2717117496 shutdown_connections 2024-06-18T02:53:35.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:35.430+0000 7f0c8c3a3640 1 -- 172.21.15.123:0/2717117496 wait complete. 2024-06-18T02:53:36.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:36 smithi016 ceph-mon[29214]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:53:36.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:36 smithi016 ceph-mon[29214]: from='client.? 172.21.15.123:0/2717117496' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-18T02:53:36.971 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-18T02:53:36.971 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph mon dump -f json 2024-06-18T02:53:37.231 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-18T02:53:37.290 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-18T02:53:38.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:38 smithi016 ceph-mon[29214]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:53:38.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:38.781+0000 7f01fa5c3640 1 -- 172.21.15.123:0/3999928666 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01f40fbf30 msgr2=0x7f01f40fc340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:53:38.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:38.781+0000 7f01fa5c3640 1 --2- 172.21.15.123:0/3999928666 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01f40fbf30 0x7f01f40fc340 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f01e4004850 tx=0x7f01e4009da0 comp rx=0 tx=0).stop 2024-06-18T02:53:38.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:38.781+0000 7f01fa5c3640 1 -- 172.21.15.123:0/3999928666 shutdown_connections 2024-06-18T02:53:38.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:38.781+0000 7f01fa5c3640 1 --2- 172.21.15.123:0/3999928666 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01f40fbf30 0x7f01f40fc340 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:53:38.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:38.781+0000 7f01fa5c3640 1 -- 172.21.15.123:0/3999928666 >> 172.21.15.123:0/3999928666 conn(0x7f01f40f7710 msgr2=0x7f01f40f9b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:53:38.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:38.781+0000 7f01fa5c3640 1 -- 172.21.15.123:0/3999928666 shutdown_connections 2024-06-18T02:53:38.790 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:38.781+0000 7f01fa5c3640 1 -- 172.21.15.123:0/3999928666 wait complete. 2024-06-18T02:53:38.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:38.782+0000 7f01fa5c3640 1 Processor -- start 2024-06-18T02:53:38.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:38.782+0000 7f01fa5c3640 1 -- start start 2024-06-18T02:53:38.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:38.782+0000 7f01fa5c3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01f40fbf30 0x7f01f40711f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:53:38.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:38.782+0000 7f01fa5c3640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01f4071730 con 0x7f01f40fbf30 2024-06-18T02:53:38.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:38.782+0000 7f01f3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01f40fbf30 0x7f01f40711f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:53:38.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:38.782+0000 7f01f3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01f40fbf30 0x7f01f40711f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.123:48378/0 (socket says 172.21.15.123:48378) 2024-06-18T02:53:38.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:38.782+0000 7f01f3fff640 1 -- 172.21.15.123:0/827289889 learned_addr learned my addr 172.21.15.123:0/827289889 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-06-18T02:53:38.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:38.782+0000 7f01f3fff640 1 -- 172.21.15.123:0/827289889 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01e4005040 con 0x7f01f40fbf30 2024-06-18T02:53:38.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:38.783+0000 7f01f3fff640 1 --2- 172.21.15.123:0/827289889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01f40fbf30 0x7f01f40711f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f01e4004820 tx=0x7f01e4009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:53:38.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:38.783+0000 7f01f0ff9640 1 -- 172.21.15.123:0/827289889 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f01e4032650 con 0x7f01f40fbf30 2024-06-18T02:53:38.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:38.783+0000 7f01fa5c3640 1 -- 172.21.15.123:0/827289889 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01f406f7c0 con 0x7f01f40fbf30 2024-06-18T02:53:38.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:38.783+0000 7f01f0ff9640 1 -- 172.21.15.123:0/827289889 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f01e4032c80 con 0x7f01f40fbf30 2024-06-18T02:53:38.791 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:38.783+0000 7f01f0ff9640 1 -- 172.21.15.123:0/827289889 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f01e403eda0 con 0x7f01f40fbf30 2024-06-18T02:53:38.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:38.783+0000 7f01fa5c3640 1 -- 172.21.15.123:0/827289889 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01f406fc90 con 0x7f01f40fbf30 2024-06-18T02:53:38.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:38.784+0000 7f01f0ff9640 1 -- 172.21.15.123:0/827289889 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 42002+0+0 (secure 0 0 0) 0x7f01e4030020 con 0x7f01f40fbf30 2024-06-18T02:53:38.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:38.784+0000 7f01fa5c3640 1 -- 172.21.15.123:0/827289889 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01f41024a0 con 0x7f01f40fbf30 2024-06-18T02:53:38.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:38.784+0000 7f01f0ff9640 1 --2- 172.21.15.123:0/827289889 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f01dc034900 0x7f01dc036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:53:38.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:38.784+0000 7f01f0ff9640 1 -- 172.21.15.123:0/827289889 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f01e40669b0 con 0x7f01f40fbf30 2024-06-18T02:53:38.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:38.784+0000 7f01f37fe640 1 --2- 172.21.15.123:0/827289889 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f01dc034900 0x7f01dc036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:53:38.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:38.784+0000 7f01f37fe640 1 --2- 172.21.15.123:0/827289889 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f01dc034900 0x7f01dc036db0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f01e8006fd0 tx=0x7f01e8006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:53:38.792 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:38.789+0000 7f01f0ff9640 1 -- 172.21.15.123:0/827289889 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f01e403c510 con 0x7f01f40fbf30 2024-06-18T02:53:39.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:39.016+0000 7f01fa5c3640 1 -- 172.21.15.123:0/827289889 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f01f405ead0 con 0x7f01f40fbf30 2024-06-18T02:53:39.016 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:39.017+0000 7f01f0ff9640 1 -- 172.21.15.123:0/827289889 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f01e403c330 con 0x7f01f40fbf30 2024-06-18T02:53:39.017 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:53:39.017 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"9e93709a-2d1d-11ef-bc9d-c7b262605968","modified":"2024-06-18T02:51:51.645370Z","created":"2024-06-18T02:51:51.645370Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi016","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-18T02:53:39.017 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-06-18T02:53:39.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:39.019+0000 7f01fa5c3640 1 -- 172.21.15.123:0/827289889 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f01dc034900 msgr2=0x7f01dc036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:53:39.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:39.019+0000 7f01fa5c3640 1 --2- 172.21.15.123:0/827289889 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f01dc034900 0x7f01dc036db0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f01e8006fd0 tx=0x7f01e8006db0 comp rx=0 tx=0).stop 2024-06-18T02:53:39.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:39.019+0000 7f01fa5c3640 1 -- 172.21.15.123:0/827289889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01f40fbf30 msgr2=0x7f01f40711f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:53:39.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:39.019+0000 7f01fa5c3640 1 --2- 172.21.15.123:0/827289889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01f40fbf30 0x7f01f40711f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f01e4004820 tx=0x7f01e4009da0 comp rx=0 tx=0).stop 2024-06-18T02:53:39.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:39.019+0000 7f01fa5c3640 1 -- 172.21.15.123:0/827289889 shutdown_connections 2024-06-18T02:53:39.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:39.019+0000 7f01fa5c3640 1 --2- 172.21.15.123:0/827289889 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f01dc034900 0x7f01dc036db0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:53:39.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:39.019+0000 7f01fa5c3640 1 --2- 172.21.15.123:0/827289889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f01f40fbf30 0x7f01f40711f0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:53:39.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:39.019+0000 7f01fa5c3640 1 -- 172.21.15.123:0/827289889 >> 172.21.15.123:0/827289889 conn(0x7f01f40f7710 msgr2=0x7f01f41030d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:53:39.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:39.019+0000 7f01fa5c3640 1 -- 172.21.15.123:0/827289889 shutdown_connections 2024-06-18T02:53:39.019 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:39.019+0000 7f01fa5c3640 1 -- 172.21.15.123:0/827289889 wait complete. 2024-06-18T02:53:39.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:39 smithi016 ceph-mon[29214]: from='client.? 172.21.15.123:0/827289889' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-18T02:53:40.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:40 smithi016 ceph-mon[29214]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:53:40.583 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-18T02:53:40.583 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph mon dump -f json 2024-06-18T02:53:40.838 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-18T02:53:40.884 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-18T02:53:42.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:42.201+0000 7f57f6bc8640 1 -- 172.21.15.123:0/1761558460 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57f00fbf30 msgr2=0x7f57f00fc340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:53:42.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:42.201+0000 7f57f6bc8640 1 --2- 172.21.15.123:0/1761558460 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57f00fbf30 0x7f57f00fc340 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f57e8004850 tx=0x7f57e8009da0 comp rx=0 tx=0).stop 2024-06-18T02:53:42.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:42.201+0000 7f57f6bc8640 1 -- 172.21.15.123:0/1761558460 shutdown_connections 2024-06-18T02:53:42.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:42.201+0000 7f57f6bc8640 1 --2- 172.21.15.123:0/1761558460 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57f00fbf30 0x7f57f00fc340 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:53:42.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:42.201+0000 7f57f6bc8640 1 -- 172.21.15.123:0/1761558460 >> 172.21.15.123:0/1761558460 conn(0x7f57f00f7710 msgr2=0x7f57f00f9b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:53:42.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:42.201+0000 7f57f6bc8640 1 -- 172.21.15.123:0/1761558460 shutdown_connections 2024-06-18T02:53:42.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:42.201+0000 7f57f6bc8640 1 -- 172.21.15.123:0/1761558460 wait complete. 2024-06-18T02:53:42.202 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:42.202+0000 7f57f6bc8640 1 Processor -- start 2024-06-18T02:53:42.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:42.202+0000 7f57f6bc8640 1 -- start start 2024-06-18T02:53:42.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:42.202+0000 7f57f6bc8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57f00fbf30 0x7f57f010e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:53:42.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:42.202+0000 7f57f6bc8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57f010ec40 con 0x7f57f00fbf30 2024-06-18T02:53:42.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:42.202+0000 7f57f4964640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57f00fbf30 0x7f57f010e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:53:42.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:42.202+0000 7f57f4964640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57f00fbf30 0x7f57f010e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.123:45048/0 (socket says 172.21.15.123:45048) 2024-06-18T02:53:42.203 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:42.202+0000 7f57f4964640 1 -- 172.21.15.123:0/381464676 learned_addr learned my addr 172.21.15.123:0/381464676 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-06-18T02:53:42.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:42.202+0000 7f57f4964640 1 -- 172.21.15.123:0/381464676 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57e8005040 con 0x7f57f00fbf30 2024-06-18T02:53:42.204 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:42.203+0000 7f57f4964640 1 --2- 172.21.15.123:0/381464676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57f00fbf30 0x7f57f010e700 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f57e8004820 tx=0x7f57e8009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:53:42.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:42.203+0000 7f57ed7fa640 1 -- 172.21.15.123:0/381464676 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57e8032650 con 0x7f57f00fbf30 2024-06-18T02:53:42.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:42.203+0000 7f57f6bc8640 1 -- 172.21.15.123:0/381464676 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57f010ee40 con 0x7f57f00fbf30 2024-06-18T02:53:42.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:42.203+0000 7f57ed7fa640 1 -- 172.21.15.123:0/381464676 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f57e8032c80 con 0x7f57f00fbf30 2024-06-18T02:53:42.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:42.203+0000 7f57ed7fa640 1 -- 172.21.15.123:0/381464676 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57e803eda0 con 0x7f57f00fbf30 2024-06-18T02:53:42.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:42.203+0000 7f57f6bc8640 1 -- 172.21.15.123:0/381464676 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57f010f9d0 con 0x7f57f00fbf30 2024-06-18T02:53:42.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:42.204+0000 7f57ed7fa640 1 -- 172.21.15.123:0/381464676 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 42002+0+0 (secure 0 0 0) 0x7f57e8030020 con 0x7f57f00fbf30 2024-06-18T02:53:42.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:42.204+0000 7f57f6bc8640 1 -- 172.21.15.123:0/381464676 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57f01024a0 con 0x7f57f00fbf30 2024-06-18T02:53:42.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:42.204+0000 7f57ed7fa640 1 --2- 172.21.15.123:0/381464676 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f57dc034900 0x7f57dc036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:53:42.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:42.204+0000 7f57ed7fa640 1 -- 172.21.15.123:0/381464676 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f57e80669b0 con 0x7f57f00fbf30 2024-06-18T02:53:42.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:42.204+0000 7f57effff640 1 --2- 172.21.15.123:0/381464676 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f57dc034900 0x7f57dc036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:53:42.205 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:42.205+0000 7f57effff640 1 --2- 172.21.15.123:0/381464676 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f57dc034900 0x7f57dc036db0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f57e000ad90 tx=0x7f57e0009410 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:53:42.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:42.210+0000 7f57ed7fa640 1 -- 172.21.15.123:0/381464676 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f57e803c510 con 0x7f57f00fbf30 2024-06-18T02:53:42.435 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:42.435+0000 7f57f6bc8640 1 -- 172.21.15.123:0/381464676 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f57f010fc10 con 0x7f57f00fbf30 2024-06-18T02:53:42.436 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:42.436+0000 7f57ed7fa640 1 -- 172.21.15.123:0/381464676 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f57e803c330 con 0x7f57f00fbf30 2024-06-18T02:53:42.436 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:53:42.436 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"9e93709a-2d1d-11ef-bc9d-c7b262605968","modified":"2024-06-18T02:51:51.645370Z","created":"2024-06-18T02:51:51.645370Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi016","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-18T02:53:42.436 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-06-18T02:53:42.437 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:42.437+0000 7f57f6bc8640 1 -- 172.21.15.123:0/381464676 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f57dc034900 msgr2=0x7f57dc036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:53:42.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:42.437+0000 7f57f6bc8640 1 --2- 172.21.15.123:0/381464676 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f57dc034900 0x7f57dc036db0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f57e000ad90 tx=0x7f57e0009410 comp rx=0 tx=0).stop 2024-06-18T02:53:42.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:42.437+0000 7f57f6bc8640 1 -- 172.21.15.123:0/381464676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57f00fbf30 msgr2=0x7f57f010e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:53:42.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:42.437+0000 7f57f6bc8640 1 --2- 172.21.15.123:0/381464676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57f00fbf30 0x7f57f010e700 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f57e8004820 tx=0x7f57e8009da0 comp rx=0 tx=0).stop 2024-06-18T02:53:42.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:42.437+0000 7f57f6bc8640 1 -- 172.21.15.123:0/381464676 shutdown_connections 2024-06-18T02:53:42.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:42.437+0000 7f57f6bc8640 1 --2- 172.21.15.123:0/381464676 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f57dc034900 0x7f57dc036db0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:53:42.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:42.437+0000 7f57f6bc8640 1 --2- 172.21.15.123:0/381464676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57f00fbf30 0x7f57f010e700 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:53:42.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:42.437+0000 7f57f6bc8640 1 -- 172.21.15.123:0/381464676 >> 172.21.15.123:0/381464676 conn(0x7f57f00f7710 msgr2=0x7f57f00f9af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:53:42.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:42.437+0000 7f57f6bc8640 1 -- 172.21.15.123:0/381464676 shutdown_connections 2024-06-18T02:53:42.438 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:42.438+0000 7f57f6bc8640 1 -- 172.21.15.123:0/381464676 wait complete. 2024-06-18T02:53:42.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:42 smithi016 ceph-mon[29214]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:53:43.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:43 smithi016 ceph-mon[29214]: from='client.? 172.21.15.123:0/381464676' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-18T02:53:43.995 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-18T02:53:43.995 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph mon dump -f json 2024-06-18T02:53:44.241 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-18T02:53:44.288 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-18T02:53:44.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:44 smithi016 ceph-mon[29214]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:53:45.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:45.561+0000 7f25933bc640 1 -- 172.21.15.123:0/2378404840 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f258c0fae70 msgr2=0x7f258c0fb280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:53:45.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:45.561+0000 7f25933bc640 1 --2- 172.21.15.123:0/2378404840 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f258c0fae70 0x7f258c0fb280 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f2584004850 tx=0x7f2584009da0 comp rx=0 tx=0).stop 2024-06-18T02:53:45.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:45.561+0000 7f25933bc640 1 -- 172.21.15.123:0/2378404840 shutdown_connections 2024-06-18T02:53:45.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:45.561+0000 7f25933bc640 1 --2- 172.21.15.123:0/2378404840 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f258c0fae70 0x7f258c0fb280 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:53:45.562 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:45.561+0000 7f25933bc640 1 -- 172.21.15.123:0/2378404840 >> 172.21.15.123:0/2378404840 conn(0x7f258c0f65e0 msgr2=0x7f258c0f8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:53:45.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:45.561+0000 7f25933bc640 1 -- 172.21.15.123:0/2378404840 shutdown_connections 2024-06-18T02:53:45.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:45.561+0000 7f25933bc640 1 -- 172.21.15.123:0/2378404840 wait complete. 2024-06-18T02:53:45.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:45.562+0000 7f25933bc640 1 Processor -- start 2024-06-18T02:53:45.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:45.562+0000 7f25933bc640 1 -- start start 2024-06-18T02:53:45.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:45.562+0000 7f25933bc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f258c0fae70 0x7f258c10c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:53:45.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:45.562+0000 7f25933bc640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f258c10cac0 con 0x7f258c0fae70 2024-06-18T02:53:45.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:45.562+0000 7f2591158640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f258c0fae70 0x7f258c10c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:53:45.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:45.562+0000 7f2591158640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f258c0fae70 0x7f258c10c580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.123:45064/0 (socket says 172.21.15.123:45064) 2024-06-18T02:53:45.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:45.562+0000 7f2591158640 1 -- 172.21.15.123:0/2467865008 learned_addr learned my addr 172.21.15.123:0/2467865008 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-06-18T02:53:45.563 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:45.562+0000 7f2591158640 1 -- 172.21.15.123:0/2467865008 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2584005040 con 0x7f258c0fae70 2024-06-18T02:53:45.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:45.563+0000 7f2591158640 1 --2- 172.21.15.123:0/2467865008 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f258c0fae70 0x7f258c10c580 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f2584004820 tx=0x7f2584009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:53:45.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:45.563+0000 7f2589ffb640 1 -- 172.21.15.123:0/2467865008 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2584032650 con 0x7f258c0fae70 2024-06-18T02:53:45.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:45.563+0000 7f2589ffb640 1 -- 172.21.15.123:0/2467865008 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2584032c80 con 0x7f258c0fae70 2024-06-18T02:53:45.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:45.563+0000 7f25933bc640 1 -- 172.21.15.123:0/2467865008 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f258c10ccc0 con 0x7f258c0fae70 2024-06-18T02:53:45.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:45.563+0000 7f2589ffb640 1 -- 172.21.15.123:0/2467865008 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2584033d60 con 0x7f258c0fae70 2024-06-18T02:53:45.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:45.563+0000 7f25933bc640 1 -- 172.21.15.123:0/2467865008 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f258c10d850 con 0x7f258c0fae70 2024-06-18T02:53:45.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:45.564+0000 7f2589ffb640 1 -- 172.21.15.123:0/2467865008 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 42002+0+0 (secure 0 0 0) 0x7f2584030020 con 0x7f258c0fae70 2024-06-18T02:53:45.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:45.564+0000 7f25933bc640 1 -- 172.21.15.123:0/2467865008 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f258c1024a0 con 0x7f258c0fae70 2024-06-18T02:53:45.565 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:45.564+0000 7f2589ffb640 1 --2- 172.21.15.123:0/2467865008 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f2574034900 0x7f2574036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:53:45.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:45.564+0000 7f2589ffb640 1 -- 172.21.15.123:0/2467865008 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f2584066750 con 0x7f258c0fae70 2024-06-18T02:53:45.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:45.565+0000 7f2590957640 1 --2- 172.21.15.123:0/2467865008 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f2574034900 0x7f2574036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:53:45.566 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:45.565+0000 7f2590957640 1 --2- 172.21.15.123:0/2467865008 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f2574034900 0x7f2574036db0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f258000ad90 tx=0x7f2580009410 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:53:45.569 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:45.569+0000 7f2589ffb640 1 -- 172.21.15.123:0/2467865008 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f2584032df0 con 0x7f258c0fae70 2024-06-18T02:53:45.793 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:45.793+0000 7f25933bc640 1 -- 172.21.15.123:0/2467865008 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f258c10e250 con 0x7f258c0fae70 2024-06-18T02:53:45.794 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:45.794+0000 7f2589ffb640 1 -- 172.21.15.123:0/2467865008 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f258403c380 con 0x7f258c0fae70 2024-06-18T02:53:45.794 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:53:45.794 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"9e93709a-2d1d-11ef-bc9d-c7b262605968","modified":"2024-06-18T02:51:51.645370Z","created":"2024-06-18T02:51:51.645370Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi016","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-18T02:53:45.795 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-06-18T02:53:45.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:45.796+0000 7f25933bc640 1 -- 172.21.15.123:0/2467865008 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f2574034900 msgr2=0x7f2574036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:53:45.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:45.796+0000 7f25933bc640 1 --2- 172.21.15.123:0/2467865008 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f2574034900 0x7f2574036db0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f258000ad90 tx=0x7f2580009410 comp rx=0 tx=0).stop 2024-06-18T02:53:45.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:45.796+0000 7f25933bc640 1 -- 172.21.15.123:0/2467865008 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f258c0fae70 msgr2=0x7f258c10c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:53:45.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:45.796+0000 7f25933bc640 1 --2- 172.21.15.123:0/2467865008 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f258c0fae70 0x7f258c10c580 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f2584004820 tx=0x7f2584009da0 comp rx=0 tx=0).stop 2024-06-18T02:53:45.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:45.796+0000 7f25933bc640 1 -- 172.21.15.123:0/2467865008 shutdown_connections 2024-06-18T02:53:45.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:45.796+0000 7f25933bc640 1 --2- 172.21.15.123:0/2467865008 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f2574034900 0x7f2574036db0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:53:45.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:45.796+0000 7f25933bc640 1 --2- 172.21.15.123:0/2467865008 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f258c0fae70 0x7f258c10c580 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:53:45.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:45.796+0000 7f25933bc640 1 -- 172.21.15.123:0/2467865008 >> 172.21.15.123:0/2467865008 conn(0x7f258c0f65e0 msgr2=0x7f258c1030d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:53:45.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:45.796+0000 7f25933bc640 1 -- 172.21.15.123:0/2467865008 shutdown_connections 2024-06-18T02:53:45.797 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:45.796+0000 7f25933bc640 1 -- 172.21.15.123:0/2467865008 wait complete. 2024-06-18T02:53:46.554 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:46 smithi016 ceph-mon[29214]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:53:46.554 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:46 smithi016 ceph-mon[29214]: from='client.? 172.21.15.123:0/2467865008' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-18T02:53:47.398 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-18T02:53:47.399 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph mon dump -f json 2024-06-18T02:53:47.646 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-18T02:53:47.692 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-18T02:53:48.305 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:48 smithi016 ceph-mon[29214]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:53:49.072 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:49.071+0000 7f067a6d7640 1 -- 172.21.15.123:0/72237901 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0674065ee0 msgr2=0x7f0674100f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:53:49.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:49.071+0000 7f067a6d7640 1 --2- 172.21.15.123:0/72237901 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0674065ee0 0x7f0674100f90 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f0664004850 tx=0x7f0664009da0 comp rx=0 tx=0).stop 2024-06-18T02:53:49.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:49.071+0000 7f067a6d7640 1 -- 172.21.15.123:0/72237901 shutdown_connections 2024-06-18T02:53:49.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:49.071+0000 7f067a6d7640 1 --2- 172.21.15.123:0/72237901 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0674065ee0 0x7f0674100f90 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:53:49.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:49.071+0000 7f067a6d7640 1 -- 172.21.15.123:0/72237901 >> 172.21.15.123:0/72237901 conn(0x7f06740f65e0 msgr2=0x7f06740f8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:53:49.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:49.071+0000 7f067a6d7640 1 -- 172.21.15.123:0/72237901 shutdown_connections 2024-06-18T02:53:49.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:49.071+0000 7f067a6d7640 1 -- 172.21.15.123:0/72237901 wait complete. 2024-06-18T02:53:49.073 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:49.072+0000 7f067a6d7640 1 Processor -- start 2024-06-18T02:53:49.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:49.072+0000 7f067a6d7640 1 -- start start 2024-06-18T02:53:49.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:49.072+0000 7f067a6d7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0674065ee0 0x7f067410e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:53:49.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:49.072+0000 7f067a6d7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f067410eca0 con 0x7f0674065ee0 2024-06-18T02:53:49.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:49.072+0000 7f0673fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0674065ee0 0x7f067410e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:53:49.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:49.072+0000 7f0673fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0674065ee0 0x7f067410e760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.123:42120/0 (socket says 172.21.15.123:42120) 2024-06-18T02:53:49.075 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:49.072+0000 7f0673fff640 1 -- 172.21.15.123:0/1464092733 learned_addr learned my addr 172.21.15.123:0/1464092733 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-06-18T02:53:49.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:49.073+0000 7f0673fff640 1 -- 172.21.15.123:0/1464092733 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0664005040 con 0x7f0674065ee0 2024-06-18T02:53:49.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:49.073+0000 7f0673fff640 1 --2- 172.21.15.123:0/1464092733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0674065ee0 0x7f067410e760 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f0664003080 tx=0x7f0664009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:53:49.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:49.073+0000 7f0670ff9640 1 -- 172.21.15.123:0/1464092733 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f06640326c0 con 0x7f0674065ee0 2024-06-18T02:53:49.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:49.073+0000 7f0670ff9640 1 -- 172.21.15.123:0/1464092733 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0664032cf0 con 0x7f0674065ee0 2024-06-18T02:53:49.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:49.073+0000 7f0670ff9640 1 -- 172.21.15.123:0/1464092733 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0664033db0 con 0x7f0674065ee0 2024-06-18T02:53:49.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:49.073+0000 7f067a6d7640 1 -- 172.21.15.123:0/1464092733 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f067410eea0 con 0x7f0674065ee0 2024-06-18T02:53:49.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:49.073+0000 7f067a6d7640 1 -- 172.21.15.123:0/1464092733 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f067410fa30 con 0x7f0674065ee0 2024-06-18T02:53:49.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:49.074+0000 7f0670ff9640 1 -- 172.21.15.123:0/1464092733 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 42002+0+0 (secure 0 0 0) 0x7f0664030020 con 0x7f0674065ee0 2024-06-18T02:53:49.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:49.074+0000 7f067a6d7640 1 -- 172.21.15.123:0/1464092733 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f067410f370 con 0x7f0674065ee0 2024-06-18T02:53:49.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:49.074+0000 7f0670ff9640 1 --2- 172.21.15.123:0/1464092733 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f065c034900 0x7f065c036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:53:49.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:49.074+0000 7f0670ff9640 1 -- 172.21.15.123:0/1464092733 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f0664066820 con 0x7f0674065ee0 2024-06-18T02:53:49.076 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:49.074+0000 7f06737fe640 1 --2- 172.21.15.123:0/1464092733 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f065c034900 0x7f065c036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:53:49.077 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:49.075+0000 7f06737fe640 1 --2- 172.21.15.123:0/1464092733 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f065c034900 0x7f065c036db0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0668006fd0 tx=0x7f0668006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:53:49.080 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:49.079+0000 7f0670ff9640 1 -- 172.21.15.123:0/1464092733 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f067410f370 con 0x7f0674065ee0 2024-06-18T02:53:49.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:49.307+0000 7f067a6d7640 1 -- 172.21.15.123:0/1464092733 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f06741046a0 con 0x7f0674065ee0 2024-06-18T02:53:49.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:49.307+0000 7f0670ff9640 1 -- 172.21.15.123:0/1464092733 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f0664033420 con 0x7f0674065ee0 2024-06-18T02:53:49.309 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:53:49.309 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"9e93709a-2d1d-11ef-bc9d-c7b262605968","modified":"2024-06-18T02:51:51.645370Z","created":"2024-06-18T02:51:51.645370Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi016","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-18T02:53:49.309 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-06-18T02:53:49.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:49.309+0000 7f067a6d7640 1 -- 172.21.15.123:0/1464092733 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f065c034900 msgr2=0x7f065c036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:53:49.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:49.309+0000 7f067a6d7640 1 --2- 172.21.15.123:0/1464092733 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f065c034900 0x7f065c036db0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0668006fd0 tx=0x7f0668006db0 comp rx=0 tx=0).stop 2024-06-18T02:53:49.310 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:49.309+0000 7f067a6d7640 1 -- 172.21.15.123:0/1464092733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0674065ee0 msgr2=0x7f067410e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:53:49.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:49.309+0000 7f067a6d7640 1 --2- 172.21.15.123:0/1464092733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0674065ee0 0x7f067410e760 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f0664003080 tx=0x7f0664009da0 comp rx=0 tx=0).stop 2024-06-18T02:53:49.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:49.309+0000 7f067a6d7640 1 -- 172.21.15.123:0/1464092733 shutdown_connections 2024-06-18T02:53:49.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:49.309+0000 7f067a6d7640 1 --2- 172.21.15.123:0/1464092733 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f065c034900 0x7f065c036db0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:53:49.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:49.309+0000 7f067a6d7640 1 --2- 172.21.15.123:0/1464092733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0674065ee0 0x7f067410e760 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:53:49.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:49.309+0000 7f067a6d7640 1 -- 172.21.15.123:0/1464092733 >> 172.21.15.123:0/1464092733 conn(0x7f06740f65e0 msgr2=0x7f06741052d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:53:49.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:49.309+0000 7f067a6d7640 1 -- 172.21.15.123:0/1464092733 shutdown_connections 2024-06-18T02:53:49.311 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:49.309+0000 7f067a6d7640 1 -- 172.21.15.123:0/1464092733 wait complete. 2024-06-18T02:53:49.576 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:49 smithi016 ceph-mon[29214]: from='mgr.14160 172.21.15.16:0/3988449102' entity='mgr.smithi016.amxydz' 2024-06-18T02:53:49.576 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:49 smithi016 ceph-mon[29214]: from='mgr.14160 172.21.15.16:0/3988449102' entity='mgr.smithi016.amxydz' 2024-06-18T02:53:49.577 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:49 smithi016 ceph-mon[29214]: from='mgr.14160 172.21.15.16:0/3988449102' entity='mgr.smithi016.amxydz' 2024-06-18T02:53:49.577 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:49 smithi016 ceph-mon[29214]: from='mgr.14160 172.21.15.16:0/3988449102' entity='mgr.smithi016.amxydz' 2024-06-18T02:53:49.577 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:49 smithi016 ceph-mon[29214]: from='mgr.14160 172.21.15.16:0/3988449102' entity='mgr.smithi016.amxydz' 2024-06-18T02:53:49.577 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:49 smithi016 ceph-mon[29214]: from='mgr.14160 172.21.15.16:0/3988449102' entity='mgr.smithi016.amxydz' 2024-06-18T02:53:49.577 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:49 smithi016 ceph-mon[29214]: Deploying daemon prometheus.smithi016 on smithi016 2024-06-18T02:53:50.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:50 smithi016 ceph-mon[29214]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:53:50.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:50 smithi016 ceph-mon[29214]: from='client.? 172.21.15.123:0/1464092733' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-18T02:53:50.915 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-18T02:53:50.915 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph mon dump -f json 2024-06-18T02:53:51.159 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-18T02:53:51.206 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-18T02:53:52.625 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:52.624+0000 7f71eefcb640 1 -- 172.21.15.123:0/728327840 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71e8065e80 msgr2=0x7f71e80febc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:53:52.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:52.624+0000 7f71eefcb640 1 --2- 172.21.15.123:0/728327840 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71e8065e80 0x7f71e80febc0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f71e0004850 tx=0x7f71e0009da0 comp rx=0 tx=0).stop 2024-06-18T02:53:52.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:52.624+0000 7f71eefcb640 1 -- 172.21.15.123:0/728327840 shutdown_connections 2024-06-18T02:53:52.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:52.624+0000 7f71eefcb640 1 --2- 172.21.15.123:0/728327840 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71e8065e80 0x7f71e80febc0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:53:52.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:52.624+0000 7f71eefcb640 1 -- 172.21.15.123:0/728327840 >> 172.21.15.123:0/728327840 conn(0x7f71e80f65f0 msgr2=0x7f71e80f8a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:53:52.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:52.624+0000 7f71eefcb640 1 -- 172.21.15.123:0/728327840 shutdown_connections 2024-06-18T02:53:52.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:52.624+0000 7f71eefcb640 1 -- 172.21.15.123:0/728327840 wait complete. 2024-06-18T02:53:52.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:52.625+0000 7f71eefcb640 1 Processor -- start 2024-06-18T02:53:52.626 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:52.625+0000 7f71eefcb640 1 -- start start 2024-06-18T02:53:52.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:52.625+0000 7f71eefcb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71e8065e80 0x7f71e810c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:53:52.627 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:52.625+0000 7f71eefcb640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71e810caa0 con 0x7f71e8065e80 2024-06-18T02:53:52.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:52.625+0000 7f71ecd67640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71e8065e80 0x7f71e810c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:53:52.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:52.625+0000 7f71ecd67640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71e8065e80 0x7f71e810c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.123:42136/0 (socket says 172.21.15.123:42136) 2024-06-18T02:53:52.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:52.626+0000 7f71ecd67640 1 -- 172.21.15.123:0/2018627424 learned_addr learned my addr 172.21.15.123:0/2018627424 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-06-18T02:53:52.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:52.626+0000 7f71ecd67640 1 -- 172.21.15.123:0/2018627424 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71e0005040 con 0x7f71e8065e80 2024-06-18T02:53:52.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:52.626+0000 7f71ecd67640 1 --2- 172.21.15.123:0/2018627424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71e8065e80 0x7f71e810c560 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f71e0004820 tx=0x7f71e0009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:53:52.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:52.626+0000 7f71e57fa640 1 -- 172.21.15.123:0/2018627424 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f71e0032650 con 0x7f71e8065e80 2024-06-18T02:53:52.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:52.626+0000 7f71eefcb640 1 -- 172.21.15.123:0/2018627424 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71e810cca0 con 0x7f71e8065e80 2024-06-18T02:53:52.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:52.626+0000 7f71e57fa640 1 -- 172.21.15.123:0/2018627424 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f71e0032c80 con 0x7f71e8065e80 2024-06-18T02:53:52.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:52.626+0000 7f71e57fa640 1 -- 172.21.15.123:0/2018627424 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f71e0033d60 con 0x7f71e8065e80 2024-06-18T02:53:52.628 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:52.626+0000 7f71eefcb640 1 -- 172.21.15.123:0/2018627424 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71e810d830 con 0x7f71e8065e80 2024-06-18T02:53:52.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:52.627+0000 7f71e57fa640 1 -- 172.21.15.123:0/2018627424 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 42002+0+0 (secure 0 0 0) 0x7f71e0030020 con 0x7f71e8065e80 2024-06-18T02:53:52.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:52.627+0000 7f71eefcb640 1 -- 172.21.15.123:0/2018627424 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71e81024b0 con 0x7f71e8065e80 2024-06-18T02:53:52.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:52.628+0000 7f71e57fa640 1 --2- 172.21.15.123:0/2018627424 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f71d4034900 0x7f71d4036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:53:52.629 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:52.628+0000 7f71e57fa640 1 -- 172.21.15.123:0/2018627424 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f71e0066750 con 0x7f71e8065e80 2024-06-18T02:53:52.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:52.628+0000 7f71e7fff640 1 --2- 172.21.15.123:0/2018627424 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f71d4034900 0x7f71d4036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:53:52.630 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:52.629+0000 7f71e7fff640 1 --2- 172.21.15.123:0/2018627424 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f71d4034900 0x7f71d4036db0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f71d800ad90 tx=0x7f71d8009410 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:53:52.634 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:52.633+0000 7f71e57fa640 1 -- 172.21.15.123:0/2018627424 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f71e0032df0 con 0x7f71e8065e80 2024-06-18T02:53:52.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:52 smithi016 ceph-mon[29214]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:53:52.861 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:52.861+0000 7f71eefcb640 1 -- 172.21.15.123:0/2018627424 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f71e810e230 con 0x7f71e8065e80 2024-06-18T02:53:52.862 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:52.861+0000 7f71e57fa640 1 -- 172.21.15.123:0/2018627424 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f71e003c380 con 0x7f71e8065e80 2024-06-18T02:53:52.863 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:53:52.863 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"9e93709a-2d1d-11ef-bc9d-c7b262605968","modified":"2024-06-18T02:51:51.645370Z","created":"2024-06-18T02:51:51.645370Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi016","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-18T02:53:52.863 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-06-18T02:53:52.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:52.863+0000 7f71eefcb640 1 -- 172.21.15.123:0/2018627424 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f71d4034900 msgr2=0x7f71d4036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:53:52.864 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:52.863+0000 7f71eefcb640 1 --2- 172.21.15.123:0/2018627424 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f71d4034900 0x7f71d4036db0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f71d800ad90 tx=0x7f71d8009410 comp rx=0 tx=0).stop 2024-06-18T02:53:52.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:52.863+0000 7f71eefcb640 1 -- 172.21.15.123:0/2018627424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71e8065e80 msgr2=0x7f71e810c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:53:52.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:52.863+0000 7f71eefcb640 1 --2- 172.21.15.123:0/2018627424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71e8065e80 0x7f71e810c560 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f71e0004820 tx=0x7f71e0009da0 comp rx=0 tx=0).stop 2024-06-18T02:53:52.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:52.863+0000 7f71eefcb640 1 -- 172.21.15.123:0/2018627424 shutdown_connections 2024-06-18T02:53:52.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:52.863+0000 7f71eefcb640 1 --2- 172.21.15.123:0/2018627424 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f71d4034900 0x7f71d4036db0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:53:52.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:52.863+0000 7f71eefcb640 1 --2- 172.21.15.123:0/2018627424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71e8065e80 0x7f71e810c560 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:53:52.865 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:52.863+0000 7f71eefcb640 1 -- 172.21.15.123:0/2018627424 >> 172.21.15.123:0/2018627424 conn(0x7f71e80f65f0 msgr2=0x7f71e81030e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:53:52.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:52.863+0000 7f71eefcb640 1 -- 172.21.15.123:0/2018627424 shutdown_connections 2024-06-18T02:53:52.866 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:52.863+0000 7f71eefcb640 1 -- 172.21.15.123:0/2018627424 wait complete. 2024-06-18T02:53:53.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:53 smithi016 ceph-mon[29214]: from='client.? 172.21.15.123:0/2018627424' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-18T02:53:53.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:53 smithi016 ceph-mon[29214]: from='mgr.14160 172.21.15.16:0/3988449102' entity='mgr.smithi016.amxydz' 2024-06-18T02:53:54.470 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-18T02:53:54.470 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph mon dump -f json 2024-06-18T02:53:54.727 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-18T02:53:54.775 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-18T02:53:54.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:54 smithi016 ceph-mon[29214]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:53:56.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:56.145+0000 7f652e30f640 1 -- 172.21.15.123:0/1088866036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65280fb040 msgr2=0x7f65280fb450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:53:56.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:56.145+0000 7f652e30f640 1 --2- 172.21.15.123:0/1088866036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65280fb040 0x7f65280fb450 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f6518004850 tx=0x7f6518009da0 comp rx=0 tx=0).stop 2024-06-18T02:53:56.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:56.145+0000 7f652e30f640 1 -- 172.21.15.123:0/1088866036 shutdown_connections 2024-06-18T02:53:56.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:56.145+0000 7f652e30f640 1 --2- 172.21.15.123:0/1088866036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65280fb040 0x7f65280fb450 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:53:56.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:56.145+0000 7f652e30f640 1 -- 172.21.15.123:0/1088866036 >> 172.21.15.123:0/1088866036 conn(0x7f65280f6820 msgr2=0x7f65280f8c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:53:56.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:56.145+0000 7f652e30f640 1 -- 172.21.15.123:0/1088866036 shutdown_connections 2024-06-18T02:53:56.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:56.145+0000 7f652e30f640 1 -- 172.21.15.123:0/1088866036 wait complete. 2024-06-18T02:53:56.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:56.145+0000 7f652e30f640 1 Processor -- start 2024-06-18T02:53:56.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:56.145+0000 7f652e30f640 1 -- start start 2024-06-18T02:53:56.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:56.146+0000 7f652e30f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65280fb040 0x7f652810c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:53:56.147 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:56.146+0000 7f652e30f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f652810cc90 con 0x7f65280fb040 2024-06-18T02:53:56.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:56.146+0000 7f6527fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65280fb040 0x7f652810c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:53:56.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:56.146+0000 7f6527fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65280fb040 0x7f652810c750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.123:42158/0 (socket says 172.21.15.123:42158) 2024-06-18T02:53:56.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:56.146+0000 7f6527fff640 1 -- 172.21.15.123:0/1438323117 learned_addr learned my addr 172.21.15.123:0/1438323117 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-06-18T02:53:56.148 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:56.146+0000 7f6527fff640 1 -- 172.21.15.123:0/1438323117 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6518005040 con 0x7f65280fb040 2024-06-18T02:53:56.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:56.146+0000 7f6527fff640 1 --2- 172.21.15.123:0/1438323117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65280fb040 0x7f652810c750 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f6518004820 tx=0x7f6518009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:53:56.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:56.146+0000 7f6524ff9640 1 -- 172.21.15.123:0/1438323117 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6518032650 con 0x7f65280fb040 2024-06-18T02:53:56.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:56.146+0000 7f6524ff9640 1 -- 172.21.15.123:0/1438323117 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6518032c80 con 0x7f65280fb040 2024-06-18T02:53:56.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:56.146+0000 7f652e30f640 1 -- 172.21.15.123:0/1438323117 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f652810ce90 con 0x7f65280fb040 2024-06-18T02:53:56.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:56.146+0000 7f6524ff9640 1 -- 172.21.15.123:0/1438323117 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f651803eda0 con 0x7f65280fb040 2024-06-18T02:53:56.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:56.146+0000 7f652e30f640 1 -- 172.21.15.123:0/1438323117 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f652810da20 con 0x7f65280fb040 2024-06-18T02:53:56.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:56.147+0000 7f6524ff9640 1 -- 172.21.15.123:0/1438323117 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 42002+0+0 (secure 0 0 0) 0x7f6518030020 con 0x7f65280fb040 2024-06-18T02:53:56.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:56.147+0000 7f652e30f640 1 -- 172.21.15.123:0/1438323117 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65281015b0 con 0x7f65280fb040 2024-06-18T02:53:56.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:56.147+0000 7f6524ff9640 1 --2- 172.21.15.123:0/1438323117 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f6510034950 0x7f6510036e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:53:56.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:56.147+0000 7f6524ff9640 1 -- 172.21.15.123:0/1438323117 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f6518066a40 con 0x7f65280fb040 2024-06-18T02:53:56.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:56.148+0000 7f65277fe640 1 --2- 172.21.15.123:0/1438323117 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f6510034950 0x7f6510036e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:53:56.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:56.148+0000 7f65277fe640 1 --2- 172.21.15.123:0/1438323117 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f6510034950 0x7f6510036e00 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f652000ad90 tx=0x7f6520009410 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:53:56.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:56.153+0000 7f6524ff9640 1 -- 172.21.15.123:0/1438323117 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f6518033420 con 0x7f65280fb040 2024-06-18T02:53:56.384 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:56.383+0000 7f652e30f640 1 -- 172.21.15.123:0/1438323117 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f65280008d0 con 0x7f65280fb040 2024-06-18T02:53:56.385 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:56.384+0000 7f6524ff9640 1 -- 172.21.15.123:0/1438323117 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f651804c300 con 0x7f65280fb040 2024-06-18T02:53:56.385 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:53:56.385 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"9e93709a-2d1d-11ef-bc9d-c7b262605968","modified":"2024-06-18T02:51:51.645370Z","created":"2024-06-18T02:51:51.645370Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi016","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-18T02:53:56.385 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-06-18T02:53:56.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:56.386+0000 7f652e30f640 1 -- 172.21.15.123:0/1438323117 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f6510034950 msgr2=0x7f6510036e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:53:56.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:56.386+0000 7f652e30f640 1 --2- 172.21.15.123:0/1438323117 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f6510034950 0x7f6510036e00 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f652000ad90 tx=0x7f6520009410 comp rx=0 tx=0).stop 2024-06-18T02:53:56.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:56.386+0000 7f652e30f640 1 -- 172.21.15.123:0/1438323117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65280fb040 msgr2=0x7f652810c750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:53:56.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:56.386+0000 7f652e30f640 1 --2- 172.21.15.123:0/1438323117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65280fb040 0x7f652810c750 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f6518004820 tx=0x7f6518009da0 comp rx=0 tx=0).stop 2024-06-18T02:53:56.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:56.386+0000 7f652e30f640 1 -- 172.21.15.123:0/1438323117 shutdown_connections 2024-06-18T02:53:56.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:56.386+0000 7f652e30f640 1 --2- 172.21.15.123:0/1438323117 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f6510034950 0x7f6510036e00 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:53:56.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:56.386+0000 7f652e30f640 1 --2- 172.21.15.123:0/1438323117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65280fb040 0x7f652810c750 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:53:56.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:56.386+0000 7f652e30f640 1 -- 172.21.15.123:0/1438323117 >> 172.21.15.123:0/1438323117 conn(0x7f65280f6820 msgr2=0x7f65280646f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:53:56.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:56.386+0000 7f652e30f640 1 -- 172.21.15.123:0/1438323117 shutdown_connections 2024-06-18T02:53:56.387 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:56.386+0000 7f652e30f640 1 -- 172.21.15.123:0/1438323117 wait complete. 2024-06-18T02:53:56.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:56 smithi016 ceph-mon[29214]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:53:57.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:57 smithi016 ceph-mon[29214]: from='client.? 172.21.15.123:0/1438323117' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-18T02:53:58.021 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-18T02:53:58.022 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph mon dump -f json 2024-06-18T02:53:58.266 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-18T02:53:58.314 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-18T02:53:58.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:53:58 smithi016 ceph-mon[29214]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:53:59.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:59.792+0000 7f09f9bf2640 1 -- 172.21.15.123:0/4092136847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09f40649d0 msgr2=0x7f09f40fa5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:53:59.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:59.792+0000 7f09f9bf2640 1 --2- 172.21.15.123:0/4092136847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09f40649d0 0x7f09f40fa5b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f09e4004850 tx=0x7f09e4009da0 comp rx=0 tx=0).stop 2024-06-18T02:53:59.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:59.792+0000 7f09f9bf2640 1 -- 172.21.15.123:0/4092136847 shutdown_connections 2024-06-18T02:53:59.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:59.792+0000 7f09f9bf2640 1 --2- 172.21.15.123:0/4092136847 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09f40649d0 0x7f09f40fa5b0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:53:59.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:59.792+0000 7f09f9bf2640 1 -- 172.21.15.123:0/4092136847 >> 172.21.15.123:0/4092136847 conn(0x7f09f40f5d70 msgr2=0x7f09f40f81a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:53:59.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:59.793+0000 7f09f9bf2640 1 -- 172.21.15.123:0/4092136847 shutdown_connections 2024-06-18T02:53:59.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:59.793+0000 7f09f9bf2640 1 -- 172.21.15.123:0/4092136847 wait complete. 2024-06-18T02:53:59.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:59.793+0000 7f09f9bf2640 1 Processor -- start 2024-06-18T02:53:59.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:59.793+0000 7f09f9bf2640 1 -- start start 2024-06-18T02:53:59.798 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:59.793+0000 7f09f9bf2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09f40649d0 0x7f09f4101710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:53:59.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:59.793+0000 7f09f9bf2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09f4101c50 con 0x7f09f40649d0 2024-06-18T02:53:59.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:59.793+0000 7f09f37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09f40649d0 0x7f09f4101710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:53:59.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:59.793+0000 7f09f37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09f40649d0 0x7f09f4101710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.123:46418/0 (socket says 172.21.15.123:46418) 2024-06-18T02:53:59.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:59.793+0000 7f09f37fe640 1 -- 172.21.15.123:0/450908131 learned_addr learned my addr 172.21.15.123:0/450908131 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-06-18T02:53:59.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:59.794+0000 7f09f37fe640 1 -- 172.21.15.123:0/450908131 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09e4005040 con 0x7f09f40649d0 2024-06-18T02:53:59.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:59.794+0000 7f09f37fe640 1 --2- 172.21.15.123:0/450908131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09f40649d0 0x7f09f4101710 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f09e4004820 tx=0x7f09e4009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:53:59.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:59.794+0000 7f09dbfff640 1 -- 172.21.15.123:0/450908131 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09e4032650 con 0x7f09f40649d0 2024-06-18T02:53:59.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:59.794+0000 7f09dbfff640 1 -- 172.21.15.123:0/450908131 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f09e4032c80 con 0x7f09f40649d0 2024-06-18T02:53:59.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:59.794+0000 7f09f9bf2640 1 -- 172.21.15.123:0/450908131 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09f40ffce0 con 0x7f09f40649d0 2024-06-18T02:53:59.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:59.794+0000 7f09dbfff640 1 -- 172.21.15.123:0/450908131 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09e403ed70 con 0x7f09f40649d0 2024-06-18T02:53:59.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:59.794+0000 7f09f9bf2640 1 -- 172.21.15.123:0/450908131 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09f41001b0 con 0x7f09f40649d0 2024-06-18T02:53:59.799 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:59.795+0000 7f09dbfff640 1 -- 172.21.15.123:0/450908131 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 42002+0+0 (secure 0 0 0) 0x7f09e4030020 con 0x7f09f40649d0 2024-06-18T02:53:59.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:59.795+0000 7f09f9bf2640 1 -- 172.21.15.123:0/450908131 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09f405ead0 con 0x7f09f40649d0 2024-06-18T02:53:59.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:59.795+0000 7f09dbfff640 1 --2- 172.21.15.123:0/450908131 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f09dc034900 0x7f09dc036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:53:59.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:59.795+0000 7f09dbfff640 1 -- 172.21.15.123:0/450908131 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f09e4066940 con 0x7f09f40649d0 2024-06-18T02:53:59.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:59.795+0000 7f09f2ffd640 1 --2- 172.21.15.123:0/450908131 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f09dc034900 0x7f09dc036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:53:59.800 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:59.796+0000 7f09f2ffd640 1 --2- 172.21.15.123:0/450908131 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f09dc034900 0x7f09dc036db0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f09e8006fd0 tx=0x7f09e8006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:53:59.801 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:53:59.800+0000 7f09dbfff640 1 -- 172.21.15.123:0/450908131 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f09e4032df0 con 0x7f09f40649d0 2024-06-18T02:54:00.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:00.024+0000 7f09f9bf2640 1 -- 172.21.15.123:0/450908131 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f09f404b480 con 0x7f09f40649d0 2024-06-18T02:54:00.026 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:00.025+0000 7f09dbfff640 1 -- 172.21.15.123:0/450908131 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f09e403c7a0 con 0x7f09f40649d0 2024-06-18T02:54:00.026 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:54:00.026 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"9e93709a-2d1d-11ef-bc9d-c7b262605968","modified":"2024-06-18T02:51:51.645370Z","created":"2024-06-18T02:51:51.645370Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi016","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-18T02:54:00.027 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-06-18T02:54:00.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:00.027+0000 7f09f9bf2640 1 -- 172.21.15.123:0/450908131 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f09dc034900 msgr2=0x7f09dc036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:54:00.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:00.027+0000 7f09f9bf2640 1 --2- 172.21.15.123:0/450908131 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f09dc034900 0x7f09dc036db0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f09e8006fd0 tx=0x7f09e8006db0 comp rx=0 tx=0).stop 2024-06-18T02:54:00.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:00.027+0000 7f09f9bf2640 1 -- 172.21.15.123:0/450908131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09f40649d0 msgr2=0x7f09f4101710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:54:00.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:00.027+0000 7f09f9bf2640 1 --2- 172.21.15.123:0/450908131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09f40649d0 0x7f09f4101710 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f09e4004820 tx=0x7f09e4009da0 comp rx=0 tx=0).stop 2024-06-18T02:54:00.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:00.027+0000 7f09f9bf2640 1 -- 172.21.15.123:0/450908131 shutdown_connections 2024-06-18T02:54:00.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:00.027+0000 7f09f9bf2640 1 --2- 172.21.15.123:0/450908131 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f09dc034900 0x7f09dc036db0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:54:00.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:00.027+0000 7f09f9bf2640 1 --2- 172.21.15.123:0/450908131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09f40649d0 0x7f09f4101710 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:54:00.028 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:00.027+0000 7f09f9bf2640 1 -- 172.21.15.123:0/450908131 >> 172.21.15.123:0/450908131 conn(0x7f09f40f5d70 msgr2=0x7f09f41030e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:54:00.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:00.027+0000 7f09f9bf2640 1 -- 172.21.15.123:0/450908131 shutdown_connections 2024-06-18T02:54:00.029 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:00.027+0000 7f09f9bf2640 1 -- 172.21.15.123:0/450908131 wait complete. 2024-06-18T02:54:00.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:00 smithi016 ceph-mon[29214]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:54:00.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:00 smithi016 ceph-mon[29214]: from='client.? 172.21.15.123:0/450908131' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-18T02:54:01.770 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-18T02:54:01.770 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph mon dump -f json 2024-06-18T02:54:02.022 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-18T02:54:02.069 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-18T02:54:02.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:02 smithi016 ceph-mon[29214]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:54:03.487 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:03.485+0000 7f0837eb6640 1 -- 172.21.15.123:0/3227595370 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08300fbf20 msgr2=0x7f08300fc330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:54:03.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:03.485+0000 7f0837eb6640 1 --2- 172.21.15.123:0/3227595370 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08300fbf20 0x7f08300fc330 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f082c009bd0 tx=0x7f082c004d10 comp rx=0 tx=0).stop 2024-06-18T02:54:03.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:03.485+0000 7f0837eb6640 1 -- 172.21.15.123:0/3227595370 shutdown_connections 2024-06-18T02:54:03.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:03.485+0000 7f0837eb6640 1 --2- 172.21.15.123:0/3227595370 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08300fbf20 0x7f08300fc330 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:54:03.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:03.485+0000 7f0837eb6640 1 -- 172.21.15.123:0/3227595370 >> 172.21.15.123:0/3227595370 conn(0x7f08300f76e0 msgr2=0x7f08300f9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:54:03.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:03.485+0000 7f0837eb6640 1 -- 172.21.15.123:0/3227595370 shutdown_connections 2024-06-18T02:54:03.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:03.485+0000 7f0837eb6640 1 -- 172.21.15.123:0/3227595370 wait complete. 2024-06-18T02:54:03.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:03.486+0000 7f0837eb6640 1 Processor -- start 2024-06-18T02:54:03.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:03.486+0000 7f0837eb6640 1 -- start start 2024-06-18T02:54:03.488 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:03.486+0000 7f0837eb6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08300fbf20 0x7f083010c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:54:03.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:03.486+0000 7f0837eb6640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f083010cad0 con 0x7f08300fbf20 2024-06-18T02:54:03.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:03.486+0000 7f0835c52640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08300fbf20 0x7f083010c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:54:03.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:03.486+0000 7f0835c52640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08300fbf20 0x7f083010c590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.123:46432/0 (socket says 172.21.15.123:46432) 2024-06-18T02:54:03.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:03.486+0000 7f0835c52640 1 -- 172.21.15.123:0/2493027622 learned_addr learned my addr 172.21.15.123:0/2493027622 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-06-18T02:54:03.489 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:03.487+0000 7f0835c52640 1 -- 172.21.15.123:0/2493027622 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f082c0098b0 con 0x7f08300fbf20 2024-06-18T02:54:03.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:03.487+0000 7f0835c52640 1 --2- 172.21.15.123:0/2493027622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08300fbf20 0x7f083010c590 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f082c009ba0 tx=0x7f082c004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:54:03.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:03.487+0000 7f08267fc640 1 -- 172.21.15.123:0/2493027622 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f082c0329b0 con 0x7f08300fbf20 2024-06-18T02:54:03.490 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:03.487+0000 7f0837eb6640 1 -- 172.21.15.123:0/2493027622 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f083010ccd0 con 0x7f08300fbf20 2024-06-18T02:54:03.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:03.487+0000 7f08267fc640 1 -- 172.21.15.123:0/2493027622 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f082c03d420 con 0x7f08300fbf20 2024-06-18T02:54:03.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:03.487+0000 7f0837eb6640 1 -- 172.21.15.123:0/2493027622 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f083010d120 con 0x7f08300fbf20 2024-06-18T02:54:03.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:03.487+0000 7f08267fc640 1 -- 172.21.15.123:0/2493027622 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f082c046560 con 0x7f08300fbf20 2024-06-18T02:54:03.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:03.488+0000 7f08267fc640 1 -- 172.21.15.123:0/2493027622 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 13) v1 ==== 42002+0+0 (secure 0 0 0) 0x7f082c02f030 con 0x7f08300fbf20 2024-06-18T02:54:03.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:03.488+0000 7f0837eb6640 1 -- 172.21.15.123:0/2493027622 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f083005ead0 con 0x7f08300fbf20 2024-06-18T02:54:03.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:03.488+0000 7f08267fc640 1 --2- 172.21.15.123:0/2493027622 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f081c0348b0 0x7f081c036d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:54:03.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:03.488+0000 7f08267fc640 1 -- 172.21.15.123:0/2493027622 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f082c067c50 con 0x7f08300fbf20 2024-06-18T02:54:03.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:03.489+0000 7f0835451640 1 --2- 172.21.15.123:0/2493027622 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f081c0348b0 0x7f081c036d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:54:03.491 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:03.489+0000 7f0835451640 1 --2- 172.21.15.123:0/2493027622 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f081c0348b0 0x7f081c036d60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0820006fd0 tx=0x7f0820006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:54:03.494 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:03.494+0000 7f08267fc640 1 -- 172.21.15.123:0/2493027622 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f082c04e920 con 0x7f08300fbf20 2024-06-18T02:54:03.721 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:03.720+0000 7f0837eb6640 1 -- 172.21.15.123:0/2493027622 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0830101cc0 con 0x7f08300fbf20 2024-06-18T02:54:03.722 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:03.721+0000 7f08267fc640 1 -- 172.21.15.123:0/2493027622 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f082c04e350 con 0x7f08300fbf20 2024-06-18T02:54:03.722 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:54:03.722 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"9e93709a-2d1d-11ef-bc9d-c7b262605968","modified":"2024-06-18T02:51:51.645370Z","created":"2024-06-18T02:51:51.645370Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi016","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-18T02:54:03.722 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-06-18T02:54:03.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:03.723+0000 7f0837eb6640 1 -- 172.21.15.123:0/2493027622 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f081c0348b0 msgr2=0x7f081c036d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:54:03.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:03.723+0000 7f0837eb6640 1 --2- 172.21.15.123:0/2493027622 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f081c0348b0 0x7f081c036d60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0820006fd0 tx=0x7f0820006db0 comp rx=0 tx=0).stop 2024-06-18T02:54:03.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:03.723+0000 7f0837eb6640 1 -- 172.21.15.123:0/2493027622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08300fbf20 msgr2=0x7f083010c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:54:03.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:03.723+0000 7f0837eb6640 1 --2- 172.21.15.123:0/2493027622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08300fbf20 0x7f083010c590 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f082c009ba0 tx=0x7f082c004d10 comp rx=0 tx=0).stop 2024-06-18T02:54:03.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:03.723+0000 7f0837eb6640 1 -- 172.21.15.123:0/2493027622 shutdown_connections 2024-06-18T02:54:03.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:03.723+0000 7f0837eb6640 1 --2- 172.21.15.123:0/2493027622 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f081c0348b0 0x7f081c036d60 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:54:03.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:03.723+0000 7f0837eb6640 1 --2- 172.21.15.123:0/2493027622 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08300fbf20 0x7f083010c590 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:54:03.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:03.723+0000 7f0837eb6640 1 -- 172.21.15.123:0/2493027622 >> 172.21.15.123:0/2493027622 conn(0x7f08300f76e0 msgr2=0x7f08301030c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:54:03.724 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:03.723+0000 7f0837eb6640 1 -- 172.21.15.123:0/2493027622 shutdown_connections 2024-06-18T02:54:03.725 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:03.723+0000 7f0837eb6640 1 -- 172.21.15.123:0/2493027622 wait complete. 2024-06-18T02:54:04.456 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:04 smithi016 ceph-mon[29214]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:54:04.457 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:04 smithi016 ceph-mon[29214]: from='client.? 172.21.15.123:0/2493027622' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-18T02:54:05.329 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-18T02:54:05.330 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph mon dump -f json 2024-06-18T02:54:05.577 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-18T02:54:05.623 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-18T02:54:06.776 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:06 smithi016 ceph-mon[29214]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:54:06.776 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:06 smithi016 ceph-mon[29214]: from='mgr.14160 172.21.15.16:0/3988449102' entity='mgr.smithi016.amxydz' 2024-06-18T02:54:06.776 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:06 smithi016 ceph-mon[29214]: from='mgr.14160 172.21.15.16:0/3988449102' entity='mgr.smithi016.amxydz' 2024-06-18T02:54:06.776 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:06 smithi016 ceph-mon[29214]: from='mgr.14160 172.21.15.16:0/3988449102' entity='mgr.smithi016.amxydz' 2024-06-18T02:54:06.776 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:06 smithi016 ceph-mon[29214]: from='mgr.14160 172.21.15.16:0/3988449102' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-06-18T02:54:06.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:06.913+0000 7f3d52094640 1 -- 172.21.15.123:0/3892205913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d4c0fbf10 msgr2=0x7f3d4c0fc320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:54:06.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:06.913+0000 7f3d52094640 1 --2- 172.21.15.123:0/3892205913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d4c0fbf10 0x7f3d4c0fc320 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3d40004850 tx=0x7f3d40009da0 comp rx=0 tx=0).stop 2024-06-18T02:54:06.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:06.913+0000 7f3d52094640 1 -- 172.21.15.123:0/3892205913 shutdown_connections 2024-06-18T02:54:06.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:06.913+0000 7f3d52094640 1 --2- 172.21.15.123:0/3892205913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d4c0fbf10 0x7f3d4c0fc320 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:54:06.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:06.913+0000 7f3d52094640 1 -- 172.21.15.123:0/3892205913 >> 172.21.15.123:0/3892205913 conn(0x7f3d4c0f76f0 msgr2=0x7f3d4c0f9b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:54:06.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:06.913+0000 7f3d52094640 1 -- 172.21.15.123:0/3892205913 shutdown_connections 2024-06-18T02:54:06.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:06.913+0000 7f3d52094640 1 -- 172.21.15.123:0/3892205913 wait complete. 2024-06-18T02:54:06.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:06.914+0000 7f3d52094640 1 Processor -- start 2024-06-18T02:54:06.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:06.914+0000 7f3d52094640 1 -- start start 2024-06-18T02:54:06.916 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:06.914+0000 7f3d52094640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d4c0fbf10 0x7f3d4c0711f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:54:06.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:06.914+0000 7f3d52094640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d4c071730 con 0x7f3d4c0fbf10 2024-06-18T02:54:06.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:06.914+0000 7f3d4b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d4c0fbf10 0x7f3d4c0711f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:54:06.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:06.914+0000 7f3d4b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d4c0fbf10 0x7f3d4c0711f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.123:46444/0 (socket says 172.21.15.123:46444) 2024-06-18T02:54:06.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:06.914+0000 7f3d4b7fe640 1 -- 172.21.15.123:0/1449470935 learned_addr learned my addr 172.21.15.123:0/1449470935 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-06-18T02:54:06.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:06.914+0000 7f3d4b7fe640 1 -- 172.21.15.123:0/1449470935 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d40005040 con 0x7f3d4c0fbf10 2024-06-18T02:54:06.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:06.915+0000 7f3d4b7fe640 1 --2- 172.21.15.123:0/1449470935 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d4c0fbf10 0x7f3d4c0711f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f3d400098d0 tx=0x7f3d40009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:54:06.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:06.915+0000 7f3d33fff640 1 -- 172.21.15.123:0/1449470935 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d40032650 con 0x7f3d4c0fbf10 2024-06-18T02:54:06.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:06.915+0000 7f3d33fff640 1 -- 172.21.15.123:0/1449470935 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3d40032c80 con 0x7f3d4c0fbf10 2024-06-18T02:54:06.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:06.915+0000 7f3d52094640 1 -- 172.21.15.123:0/1449470935 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d4c06f7c0 con 0x7f3d4c0fbf10 2024-06-18T02:54:06.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:06.915+0000 7f3d33fff640 1 -- 172.21.15.123:0/1449470935 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d40033c10 con 0x7f3d4c0fbf10 2024-06-18T02:54:06.917 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:06.915+0000 7f3d52094640 1 -- 172.21.15.123:0/1449470935 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d4c06fc30 con 0x7f3d4c0fbf10 2024-06-18T02:54:06.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:06.916+0000 7f3d33fff640 1 -- 172.21.15.123:0/1449470935 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 14) v1 ==== 42016+0+0 (secure 0 0 0) 0x7f3d40030020 con 0x7f3d4c0fbf10 2024-06-18T02:54:06.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:06.916+0000 7f3d33fff640 1 --2- 172.21.15.123:0/1449470935 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f3d34034950 0x7f3d34036e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:54:06.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:06.916+0000 7f3d52094640 1 -- 172.21.15.123:0/1449470935 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d4c102480 con 0x7f3d4c0fbf10 2024-06-18T02:54:06.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:06.916+0000 7f3d33fff640 1 -- 172.21.15.123:0/1449470935 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f3d40066a80 con 0x7f3d4c0fbf10 2024-06-18T02:54:06.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:06.916+0000 7f3d4affd640 1 -- 172.21.15.123:0/1449470935 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f3d34034950 msgr2=0x7f3d34036e00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/3749129160 2024-06-18T02:54:06.918 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:06.916+0000 7f3d4affd640 1 --2- 172.21.15.123:0/1449470935 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f3d34034950 0x7f3d34036e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-18T02:54:06.920 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:06.921+0000 7f3d33fff640 1 -- 172.21.15.123:0/1449470935 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f3d400302d0 con 0x7f3d4c0fbf10 2024-06-18T02:54:07.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:07.117+0000 7f3d4affd640 1 -- 172.21.15.123:0/1449470935 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f3d34034950 msgr2=0x7f3d34036e00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/3749129160 2024-06-18T02:54:07.117 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:07.117+0000 7f3d4affd640 1 --2- 172.21.15.123:0/1449470935 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f3d34034950 0x7f3d34036e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-18T02:54:07.149 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:07.150+0000 7f3d52094640 1 -- 172.21.15.123:0/1449470935 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3d4c05ead0 con 0x7f3d4c0fbf10 2024-06-18T02:54:07.150 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:07.150+0000 7f3d33fff640 1 -- 172.21.15.123:0/1449470935 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f3d4003aac0 con 0x7f3d4c0fbf10 2024-06-18T02:54:07.150 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:54:07.150 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"9e93709a-2d1d-11ef-bc9d-c7b262605968","modified":"2024-06-18T02:51:51.645370Z","created":"2024-06-18T02:51:51.645370Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi016","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-18T02:54:07.151 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-06-18T02:54:07.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:07.153+0000 7f3d52094640 1 -- 172.21.15.123:0/1449470935 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f3d34034950 msgr2=0x7f3d34036e00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-18T02:54:07.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:07.153+0000 7f3d52094640 1 --2- 172.21.15.123:0/1449470935 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f3d34034950 0x7f3d34036e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:54:07.153 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:07.153+0000 7f3d52094640 1 -- 172.21.15.123:0/1449470935 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d4c0fbf10 msgr2=0x7f3d4c0711f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:54:07.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:07.153+0000 7f3d52094640 1 --2- 172.21.15.123:0/1449470935 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d4c0fbf10 0x7f3d4c0711f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f3d400098d0 tx=0x7f3d40009da0 comp rx=0 tx=0).stop 2024-06-18T02:54:07.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:07.153+0000 7f3d52094640 1 -- 172.21.15.123:0/1449470935 shutdown_connections 2024-06-18T02:54:07.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:07.153+0000 7f3d52094640 1 --2- 172.21.15.123:0/1449470935 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7f3d34034950 0x7f3d34036e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:54:07.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:07.153+0000 7f3d52094640 1 --2- 172.21.15.123:0/1449470935 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d4c0fbf10 0x7f3d4c0711f0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:54:07.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:07.153+0000 7f3d52094640 1 -- 172.21.15.123:0/1449470935 >> 172.21.15.123:0/1449470935 conn(0x7f3d4c0f76f0 msgr2=0x7f3d4c1030b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:54:07.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:07.153+0000 7f3d52094640 1 -- 172.21.15.123:0/1449470935 shutdown_connections 2024-06-18T02:54:07.154 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:07.154+0000 7f3d52094640 1 -- 172.21.15.123:0/1449470935 wait complete. 2024-06-18T02:54:07.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:07 smithi016 ceph-mon[29214]: from='mgr.14160 172.21.15.16:0/3988449102' entity='mgr.smithi016.amxydz' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-06-18T02:54:07.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:07 smithi016 ceph-mon[29214]: mgrmap e14: smithi016.amxydz(active, since 68s) 2024-06-18T02:54:07.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:07 smithi016 ceph-mon[29214]: from='client.? 172.21.15.123:0/1449470935' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-18T02:54:08.759 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-18T02:54:08.760 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph mon dump -f json 2024-06-18T02:54:09.001 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-18T02:54:09.048 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-18T02:54:10.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:10.434+0000 7ff59f0a2640 1 -- 172.21.15.123:0/3669228771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5980fbf30 msgr2=0x7ff5980fc340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:54:10.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:10.434+0000 7ff59f0a2640 1 --2- 172.21.15.123:0/3669228771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5980fbf30 0x7ff5980fc340 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff58c004850 tx=0x7ff58c009da0 comp rx=0 tx=0).stop 2024-06-18T02:54:10.440 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:10.434+0000 7ff59f0a2640 1 -- 172.21.15.123:0/3669228771 shutdown_connections 2024-06-18T02:54:10.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:10.434+0000 7ff59f0a2640 1 --2- 172.21.15.123:0/3669228771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5980fbf30 0x7ff5980fc340 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:54:10.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:10.434+0000 7ff59f0a2640 1 -- 172.21.15.123:0/3669228771 >> 172.21.15.123:0/3669228771 conn(0x7ff5980f7710 msgr2=0x7ff5980f9b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:54:10.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:10.434+0000 7ff59f0a2640 1 -- 172.21.15.123:0/3669228771 shutdown_connections 2024-06-18T02:54:10.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:10.434+0000 7ff59f0a2640 1 -- 172.21.15.123:0/3669228771 wait complete. 2024-06-18T02:54:10.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:10.434+0000 7ff59f0a2640 1 Processor -- start 2024-06-18T02:54:10.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:10.435+0000 7ff59f0a2640 1 -- start start 2024-06-18T02:54:10.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:10.435+0000 7ff59f0a2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5980fbf30 0x7ff59810c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:54:10.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:10.435+0000 7ff59f0a2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff59810ca70 con 0x7ff5980fbf30 2024-06-18T02:54:10.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:10.435+0000 7ff59ce3e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5980fbf30 0x7ff59810c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:54:10.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:10.435+0000 7ff59ce3e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5980fbf30 0x7ff59810c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.123:51070/0 (socket says 172.21.15.123:51070) 2024-06-18T02:54:10.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:10.435+0000 7ff59ce3e640 1 -- 172.21.15.123:0/826053121 learned_addr learned my addr 172.21.15.123:0/826053121 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-06-18T02:54:10.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:10.435+0000 7ff59ce3e640 1 -- 172.21.15.123:0/826053121 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff58c005040 con 0x7ff5980fbf30 2024-06-18T02:54:10.441 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:10.435+0000 7ff59ce3e640 1 --2- 172.21.15.123:0/826053121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5980fbf30 0x7ff59810c530 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff58c004820 tx=0x7ff58c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:54:10.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:10.436+0000 7ff5957fa640 1 -- 172.21.15.123:0/826053121 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff58c032650 con 0x7ff5980fbf30 2024-06-18T02:54:10.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:10.436+0000 7ff5957fa640 1 -- 172.21.15.123:0/826053121 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff58c032c80 con 0x7ff5980fbf30 2024-06-18T02:54:10.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:10.436+0000 7ff59f0a2640 1 -- 172.21.15.123:0/826053121 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff59810cc70 con 0x7ff5980fbf30 2024-06-18T02:54:10.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:10.436+0000 7ff5957fa640 1 -- 172.21.15.123:0/826053121 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff58c03eda0 con 0x7ff5980fbf30 2024-06-18T02:54:10.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:10.436+0000 7ff59f0a2640 1 -- 172.21.15.123:0/826053121 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff59810d800 con 0x7ff5980fbf30 2024-06-18T02:54:10.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:10.436+0000 7ff5957fa640 1 -- 172.21.15.123:0/826053121 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 14) v1 ==== 42016+0+0 (secure 0 0 0) 0x7ff58c030020 con 0x7ff5980fbf30 2024-06-18T02:54:10.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:10.437+0000 7ff5957fa640 1 --2- 172.21.15.123:0/826053121 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7ff5880349a0 0x7ff588036e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:54:10.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:10.437+0000 7ff59f0a2640 1 -- 172.21.15.123:0/826053121 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5981024a0 con 0x7ff5980fbf30 2024-06-18T02:54:10.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:10.437+0000 7ff5957fa640 1 -- 172.21.15.123:0/826053121 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7ff58c066980 con 0x7ff5980fbf30 2024-06-18T02:54:10.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:10.437+0000 7ff597fff640 1 -- 172.21.15.123:0/826053121 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7ff5880349a0 msgr2=0x7ff588036e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/3749129160 2024-06-18T02:54:10.442 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:10.437+0000 7ff597fff640 1 --2- 172.21.15.123:0/826053121 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7ff5880349a0 0x7ff588036e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-18T02:54:10.443 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:10.441+0000 7ff5957fa640 1 -- 172.21.15.123:0/826053121 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7ff58c0327b0 con 0x7ff5980fbf30 2024-06-18T02:54:10.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:10.638+0000 7ff597fff640 1 -- 172.21.15.123:0/826053121 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7ff5880349a0 msgr2=0x7ff588036e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.16:6800/3749129160 2024-06-18T02:54:10.638 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:10.638+0000 7ff597fff640 1 --2- 172.21.15.123:0/826053121 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7ff5880349a0 0x7ff588036e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-18T02:54:10.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:10.667+0000 7ff59f0a2640 1 -- 172.21.15.123:0/826053121 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff59804b480 con 0x7ff5980fbf30 2024-06-18T02:54:10.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:10.667+0000 7ff5957fa640 1 -- 172.21.15.123:0/826053121 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7ff58c03c330 con 0x7ff5980fbf30 2024-06-18T02:54:10.667 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:54:10.667 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"9e93709a-2d1d-11ef-bc9d-c7b262605968","modified":"2024-06-18T02:51:51.645370Z","created":"2024-06-18T02:51:51.645370Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi016","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-18T02:54:10.668 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-06-18T02:54:10.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:10.669+0000 7ff59f0a2640 1 -- 172.21.15.123:0/826053121 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7ff5880349a0 msgr2=0x7ff588036e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-18T02:54:10.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:10.669+0000 7ff59f0a2640 1 --2- 172.21.15.123:0/826053121 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7ff5880349a0 0x7ff588036e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:54:10.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:10.669+0000 7ff59f0a2640 1 -- 172.21.15.123:0/826053121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5980fbf30 msgr2=0x7ff59810c530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:54:10.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:10.669+0000 7ff59f0a2640 1 --2- 172.21.15.123:0/826053121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5980fbf30 0x7ff59810c530 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff58c004820 tx=0x7ff58c009da0 comp rx=0 tx=0).stop 2024-06-18T02:54:10.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:10.669+0000 7ff59f0a2640 1 -- 172.21.15.123:0/826053121 shutdown_connections 2024-06-18T02:54:10.669 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:10.669+0000 7ff59f0a2640 1 --2- 172.21.15.123:0/826053121 >> [v2:172.21.15.16:6800/3749129160,v1:172.21.15.16:6801/3749129160] conn(0x7ff5880349a0 0x7ff588036e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:54:10.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:10.669+0000 7ff59f0a2640 1 --2- 172.21.15.123:0/826053121 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5980fbf30 0x7ff59810c530 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:54:10.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:10.669+0000 7ff59f0a2640 1 -- 172.21.15.123:0/826053121 >> 172.21.15.123:0/826053121 conn(0x7ff5980f7710 msgr2=0x7ff5981030d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:54:10.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:10.669+0000 7ff59f0a2640 1 -- 172.21.15.123:0/826053121 shutdown_connections 2024-06-18T02:54:10.670 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:10.669+0000 7ff59f0a2640 1 -- 172.21.15.123:0/826053121 wait complete. 2024-06-18T02:54:11.040 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:10 smithi016 ceph-mon[29214]: from='client.? 172.21.15.123:0/826053121' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-18T02:54:12.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:11 smithi016 ceph-mon[29214]: Active manager daemon smithi016.amxydz restarted 2024-06-18T02:54:12.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:11 smithi016 ceph-mon[29214]: Activating manager daemon smithi016.amxydz 2024-06-18T02:54:12.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:11 smithi016 ceph-mon[29214]: osdmap e5: 0 total, 0 up, 0 in 2024-06-18T02:54:12.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:11 smithi016 ceph-mon[29214]: mgrmap e15: smithi016.amxydz(active, starting, since 0.00302635s) 2024-06-18T02:54:12.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:11 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "mon metadata", "id": "smithi016"}]: dispatch 2024-06-18T02:54:12.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:11 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "mgr metadata", "who": "smithi016.amxydz", "id": "smithi016.amxydz"}]: dispatch 2024-06-18T02:54:12.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:11 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-18T02:54:12.056 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:11 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-18T02:54:12.056 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:11 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-18T02:54:12.056 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:11 smithi016 ceph-mon[29214]: Manager daemon smithi016.amxydz is now available 2024-06-18T02:54:12.056 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:11 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:54:12.056 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:11 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:54:12.056 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:11 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:54:12.056 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:11 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:54:12.056 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:11 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:54:12.056 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:11 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi016.amxydz/mirror_snapshot_schedule"}]: dispatch 2024-06-18T02:54:12.056 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:11 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi016.amxydz/trash_purge_schedule"}]: dispatch 2024-06-18T02:54:12.254 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-18T02:54:12.254 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph mon dump -f json 2024-06-18T02:54:12.505 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-18T02:54:12.553 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-18T02:54:13.158 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:12 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:54:13.158 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:12 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:54:13.158 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:12 smithi016 ceph-mon[29214]: mgrmap e16: smithi016.amxydz(active, since 1.00498s) 2024-06-18T02:54:13.158 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:12 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:54:14.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:14.064+0000 7fb58d134640 1 -- 172.21.15.123:0/2569828304 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5880fbeb0 msgr2=0x7fb5880fc2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:54:14.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:14.064+0000 7fb58d134640 1 --2- 172.21.15.123:0/2569828304 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5880fbeb0 0x7fb5880fc2c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb578004850 tx=0x7fb578009da0 comp rx=0 tx=0).stop 2024-06-18T02:54:14.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:14.064+0000 7fb58d134640 1 -- 172.21.15.123:0/2569828304 shutdown_connections 2024-06-18T02:54:14.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:14.064+0000 7fb58d134640 1 --2- 172.21.15.123:0/2569828304 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5880fbeb0 0x7fb5880fc2c0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:54:14.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:14.064+0000 7fb58d134640 1 -- 172.21.15.123:0/2569828304 >> 172.21.15.123:0/2569828304 conn(0x7fb5880f7670 msgr2=0x7fb5880f9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:54:14.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:14.064+0000 7fb58d134640 1 -- 172.21.15.123:0/2569828304 shutdown_connections 2024-06-18T02:54:14.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:14.064+0000 7fb58d134640 1 -- 172.21.15.123:0/2569828304 wait complete. 2024-06-18T02:54:14.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:14.065+0000 7fb58d134640 1 Processor -- start 2024-06-18T02:54:14.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:14.065+0000 7fb58d134640 1 -- start start 2024-06-18T02:54:14.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:14.065+0000 7fb58d134640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5880fbeb0 0x7fb5880711f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:54:14.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:14.065+0000 7fb58d134640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb588071730 con 0x7fb5880fbeb0 2024-06-18T02:54:14.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:14.065+0000 7fb586d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5880fbeb0 0x7fb5880711f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:54:14.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:14.065+0000 7fb586d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5880fbeb0 0x7fb5880711f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.123:51078/0 (socket says 172.21.15.123:51078) 2024-06-18T02:54:14.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:14.065+0000 7fb586d9d640 1 -- 172.21.15.123:0/731429809 learned_addr learned my addr 172.21.15.123:0/731429809 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-06-18T02:54:14.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:14.066+0000 7fb586d9d640 1 -- 172.21.15.123:0/731429809 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb578005040 con 0x7fb5880fbeb0 2024-06-18T02:54:14.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:14.066+0000 7fb586d9d640 1 --2- 172.21.15.123:0/731429809 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5880fbeb0 0x7fb5880711f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb578004820 tx=0x7fb578009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:54:14.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:14.066+0000 7fb57ffff640 1 -- 172.21.15.123:0/731429809 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb578032650 con 0x7fb5880fbeb0 2024-06-18T02:54:14.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:14.066+0000 7fb58d134640 1 -- 172.21.15.123:0/731429809 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb58806f7c0 con 0x7fb5880fbeb0 2024-06-18T02:54:14.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:14.066+0000 7fb57ffff640 1 -- 172.21.15.123:0/731429809 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb578032c80 con 0x7fb5880fbeb0 2024-06-18T02:54:14.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:14.066+0000 7fb58d134640 1 -- 172.21.15.123:0/731429809 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb58806fc90 con 0x7fb5880fbeb0 2024-06-18T02:54:14.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:14.066+0000 7fb57ffff640 1 -- 172.21.15.123:0/731429809 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb57803ed40 con 0x7fb5880fbeb0 2024-06-18T02:54:14.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:14.067+0000 7fb57ffff640 1 -- 172.21.15.123:0/731429809 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) v1 ==== 42059+0+0 (secure 0 0 0) 0x7fb578030020 con 0x7fb5880fbeb0 2024-06-18T02:54:14.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:14.067+0000 7fb58d134640 1 -- 172.21.15.123:0/731429809 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb58805ead0 con 0x7fb5880fbeb0 2024-06-18T02:54:14.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:14.067+0000 7fb57ffff640 1 --2- 172.21.15.123:0/731429809 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fb5740349f0 0x7fb574036ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:54:14.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:14.067+0000 7fb57ffff640 1 -- 172.21.15.123:0/731429809 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fb578066bc0 con 0x7fb5880fbeb0 2024-06-18T02:54:14.068 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:14.067+0000 7fb57e59c640 1 --2- 172.21.15.123:0/731429809 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fb5740349f0 0x7fb574036ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:54:14.069 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:14.068+0000 7fb57e59c640 1 --2- 172.21.15.123:0/731429809 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fb5740349f0 0x7fb574036ea0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb570006fd0 tx=0x7fb570006db0 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:54:14.074 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:14.074+0000 7fb57ffff640 1 -- 172.21.15.123:0/731429809 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fb5780327b0 con 0x7fb5880fbeb0 2024-06-18T02:54:14.305 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:14.305+0000 7fb58d134640 1 -- 172.21.15.123:0/731429809 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb5881057c0 con 0x7fb5880fbeb0 2024-06-18T02:54:14.305 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:14.305+0000 7fb57ffff640 1 -- 172.21.15.123:0/731429809 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fb57803c330 con 0x7fb5880fbeb0 2024-06-18T02:54:14.306 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:54:14.306 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"9e93709a-2d1d-11ef-bc9d-c7b262605968","modified":"2024-06-18T02:51:51.645370Z","created":"2024-06-18T02:51:51.645370Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi016","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-18T02:54:14.306 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-06-18T02:54:14.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:14.307+0000 7fb58d134640 1 -- 172.21.15.123:0/731429809 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fb5740349f0 msgr2=0x7fb574036ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:54:14.307 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:14.307+0000 7fb58d134640 1 --2- 172.21.15.123:0/731429809 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fb5740349f0 0x7fb574036ea0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb570006fd0 tx=0x7fb570006db0 comp rx=0 tx=0).stop 2024-06-18T02:54:14.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:14.307+0000 7fb58d134640 1 -- 172.21.15.123:0/731429809 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5880fbeb0 msgr2=0x7fb5880711f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:54:14.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:14.307+0000 7fb58d134640 1 --2- 172.21.15.123:0/731429809 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5880fbeb0 0x7fb5880711f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb578004820 tx=0x7fb578009da0 comp rx=0 tx=0).stop 2024-06-18T02:54:14.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:14.307+0000 7fb58d134640 1 -- 172.21.15.123:0/731429809 shutdown_connections 2024-06-18T02:54:14.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:14.307+0000 7fb58d134640 1 --2- 172.21.15.123:0/731429809 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fb5740349f0 0x7fb574036ea0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:54:14.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:14.307+0000 7fb58d134640 1 --2- 172.21.15.123:0/731429809 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5880fbeb0 0x7fb5880711f0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:54:14.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:14.307+0000 7fb58d134640 1 -- 172.21.15.123:0/731429809 >> 172.21.15.123:0/731429809 conn(0x7fb5880f7670 msgr2=0x7fb5880f8210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:54:14.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:14.308+0000 7fb58d134640 1 -- 172.21.15.123:0/731429809 shutdown_connections 2024-06-18T02:54:14.308 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:14.308+0000 7fb58d134640 1 -- 172.21.15.123:0/731429809 wait complete. 2024-06-18T02:54:14.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:14 smithi016 ceph-mon[29214]: [18/Jun/2024:02:54:13] ENGINE Bus STARTING 2024-06-18T02:54:14.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:14 smithi016 ceph-mon[29214]: [18/Jun/2024:02:54:13] ENGINE Serving on https://172.21.15.16:7150 2024-06-18T02:54:14.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:14 smithi016 ceph-mon[29214]: [18/Jun/2024:02:54:13] ENGINE Bus STARTED 2024-06-18T02:54:14.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:14 smithi016 ceph-mon[29214]: [18/Jun/2024:02:54:13] ENGINE Error in HTTPServer.serve 2024-06-18T02:54:14.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: Traceback (most recent call last): 2024-06-18T02:54:14.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-06-18T02:54:14.806 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: self._connections.run(self.expiration_interval) 2024-06-18T02:54:14.806 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-06-18T02:54:14.806 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: self._run(expiration_interval) 2024-06-18T02:54:14.806 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-06-18T02:54:14.806 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-06-18T02:54:14.806 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-06-18T02:54:14.806 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-06-18T02:54:14.806 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-06-18T02:54:14.806 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: s = self.context.wrap_socket( 2024-06-18T02:54:14.806 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-06-18T02:54:14.806 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: return self.sslsocket_class._create( 2024-06-18T02:54:14.806 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-06-18T02:54:14.807 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: self.do_handshake() 2024-06-18T02:54:14.807 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-06-18T02:54:14.807 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: self._sslobj.do_handshake() 2024-06-18T02:54:14.807 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1129) 2024-06-18T02:54:14.807 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:14 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:54:14.807 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:14 smithi016 ceph-mon[29214]: mgrmap e17: smithi016.amxydz(active, since 2s) 2024-06-18T02:54:14.807 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:14 smithi016 ceph-mon[29214]: from='client.? 172.21.15.123:0/731429809' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-18T02:54:16.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:16 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:54:16.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:16 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:54:16.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:16 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:54:16.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:16 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:54:16.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:16 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config rm", "who": "osd/host:smithi016", "name": "osd_memory_target"}]: dispatch 2024-06-18T02:54:17.145 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-18T02:54:17.146 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph mon dump -f json 2024-06-18T02:54:17.396 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-18T02:54:18.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:17 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:54:18.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:17 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:54:18.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:17 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:54:18.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:17 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:54:18.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:17 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config rm", "who": "osd/host:smithi123", "name": "osd_memory_target"}]: dispatch 2024-06-18T02:54:18.056 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:17 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:54:18.056 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:17 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T02:54:18.056 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:17 smithi016 ceph-mon[29214]: Updating smithi016:/etc/ceph/ceph.conf 2024-06-18T02:54:18.056 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:17 smithi016 ceph-mon[29214]: Updating smithi123:/etc/ceph/ceph.conf 2024-06-18T02:54:18.056 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:17 smithi016 ceph-mon[29214]: Updating smithi123:/var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-18T02:54:18.056 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:17 smithi016 ceph-mon[29214]: Updating smithi016:/var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-18T02:54:18.056 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:17 smithi016 ceph-mon[29214]: Updating smithi123:/etc/ceph/ceph.client.admin.keyring 2024-06-18T02:54:18.056 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:17 smithi016 ceph-mon[29214]: Updating smithi016:/etc/ceph/ceph.client.admin.keyring 2024-06-18T02:54:18.056 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:17 smithi016 ceph-mon[29214]: Updating smithi123:/var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/config/ceph.client.admin.keyring 2024-06-18T02:54:18.056 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:17 smithi016 ceph-mon[29214]: Updating smithi016:/var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/config/ceph.client.admin.keyring 2024-06-18T02:54:18.056 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:17 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:54:18.056 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:17 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:54:18.056 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:17 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:54:18.057 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:17 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:54:18.057 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:17 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:54:18.057 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:17 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi123", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-06-18T02:54:18.057 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:17 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi123", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-06-18T02:54:18.057 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:17 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:54:19.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:18 smithi016 ceph-mon[29214]: Deploying daemon crash.smithi123 on smithi123 2024-06-18T02:54:20.210 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:20.208+0000 7f334a63f640 1 -- 172.21.15.123:0/29080751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3344100e40 msgr2=0x7f3344101210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:54:20.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:20.208+0000 7f334a63f640 1 --2- 172.21.15.123:0/29080751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3344100e40 0x7f3344101210 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f3334004850 tx=0x7f3334009da0 comp rx=0 tx=0).stop 2024-06-18T02:54:20.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:20.208+0000 7f334a63f640 1 -- 172.21.15.123:0/29080751 shutdown_connections 2024-06-18T02:54:20.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:20.208+0000 7f334a63f640 1 --2- 172.21.15.123:0/29080751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3344100e40 0x7f3344101210 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:54:20.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:20.208+0000 7f334a63f640 1 -- 172.21.15.123:0/29080751 >> 172.21.15.123:0/29080751 conn(0x7f3344070e10 msgr2=0x7f3344071210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:54:20.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:20.208+0000 7f334a63f640 1 -- 172.21.15.123:0/29080751 shutdown_connections 2024-06-18T02:54:20.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:20.208+0000 7f334a63f640 1 -- 172.21.15.123:0/29080751 wait complete. 2024-06-18T02:54:20.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:20.209+0000 7f334a63f640 1 Processor -- start 2024-06-18T02:54:20.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:20.209+0000 7f334a63f640 1 -- start start 2024-06-18T02:54:20.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:20.209+0000 7f334a63f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3344100e40 0x7f334410d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:54:20.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:20.209+0000 7f334a63f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33440656a0 con 0x7f3344100e40 2024-06-18T02:54:20.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:20.210+0000 7f3343fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3344100e40 0x7f334410d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:54:20.211 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:20.210+0000 7f3343fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3344100e40 0x7f334410d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.123:52812/0 (socket says 172.21.15.123:52812) 2024-06-18T02:54:20.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:20.210+0000 7f3343fff640 1 -- 172.21.15.123:0/1397054142 learned_addr learned my addr 172.21.15.123:0/1397054142 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-06-18T02:54:20.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:20.210+0000 7f3343fff640 1 -- 172.21.15.123:0/1397054142 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3334005040 con 0x7f3344100e40 2024-06-18T02:54:20.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:20.210+0000 7f3343fff640 1 --2- 172.21.15.123:0/1397054142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3344100e40 0x7f334410d690 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f3334000c00 tx=0x7f3334009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:54:20.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:20.210+0000 7f33417fa640 1 -- 172.21.15.123:0/1397054142 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3334033700 con 0x7f3344100e40 2024-06-18T02:54:20.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:20.210+0000 7f33417fa640 1 -- 172.21.15.123:0/1397054142 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3334033d30 con 0x7f3344100e40 2024-06-18T02:54:20.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:20.210+0000 7f33417fa640 1 -- 172.21.15.123:0/1397054142 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f333403d390 con 0x7f3344100e40 2024-06-18T02:54:20.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:20.210+0000 7f334a63f640 1 -- 172.21.15.123:0/1397054142 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f334410dc30 con 0x7f3344100e40 2024-06-18T02:54:20.212 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:20.211+0000 7f334a63f640 1 -- 172.21.15.123:0/1397054142 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f334410e100 con 0x7f3344100e40 2024-06-18T02:54:20.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:20.211+0000 7f33417fa640 1 -- 172.21.15.123:0/1397054142 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) v1 ==== 42059+0+0 (secure 0 0 0) 0x7f3334030020 con 0x7f3344100e40 2024-06-18T02:54:20.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:20.212+0000 7f33417fa640 1 --2- 172.21.15.123:0/1397054142 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f3330034a40 0x7f3330036ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:54:20.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:20.212+0000 7f33417fa640 1 -- 172.21.15.123:0/1397054142 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f3334067ae0 con 0x7f3344100e40 2024-06-18T02:54:20.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:20.212+0000 7f334a63f640 1 -- 172.21.15.123:0/1397054142 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33441034f0 con 0x7f3344100e40 2024-06-18T02:54:20.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:20.212+0000 7f333bfff640 1 --2- 172.21.15.123:0/1397054142 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f3330034a40 0x7f3330036ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:54:20.213 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:20.212+0000 7f333bfff640 1 --2- 172.21.15.123:0/1397054142 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f3330034a40 0x7f3330036ef0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f332c006fd0 tx=0x7f332c006db0 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:54:20.217 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:20.217+0000 7f33417fa640 1 -- 172.21.15.123:0/1397054142 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f33340458e0 con 0x7f3344100e40 2024-06-18T02:54:20.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:20.456+0000 7f334a63f640 1 -- 172.21.15.123:0/1397054142 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3344105250 con 0x7f3344100e40 2024-06-18T02:54:20.456 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:20.457+0000 7f33417fa640 1 -- 172.21.15.123:0/1397054142 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f33340452f0 con 0x7f3344100e40 2024-06-18T02:54:20.457 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:54:20.457 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"9e93709a-2d1d-11ef-bc9d-c7b262605968","modified":"2024-06-18T02:51:51.645370Z","created":"2024-06-18T02:51:51.645370Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi016","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-18T02:54:20.457 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-06-18T02:54:20.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:20.459+0000 7f334a63f640 1 -- 172.21.15.123:0/1397054142 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f3330034a40 msgr2=0x7f3330036ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:54:20.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:20.459+0000 7f334a63f640 1 --2- 172.21.15.123:0/1397054142 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f3330034a40 0x7f3330036ef0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f332c006fd0 tx=0x7f332c006db0 comp rx=0 tx=0).stop 2024-06-18T02:54:20.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:20.459+0000 7f334a63f640 1 -- 172.21.15.123:0/1397054142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3344100e40 msgr2=0x7f334410d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:54:20.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:20.459+0000 7f334a63f640 1 --2- 172.21.15.123:0/1397054142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3344100e40 0x7f334410d690 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f3334000c00 tx=0x7f3334009da0 comp rx=0 tx=0).stop 2024-06-18T02:54:20.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:20.459+0000 7f334a63f640 1 -- 172.21.15.123:0/1397054142 shutdown_connections 2024-06-18T02:54:20.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:20.459+0000 7f334a63f640 1 --2- 172.21.15.123:0/1397054142 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f3330034a40 0x7f3330036ef0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:54:20.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:20.459+0000 7f334a63f640 1 --2- 172.21.15.123:0/1397054142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3344100e40 0x7f334410d690 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:54:20.459 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:20.459+0000 7f334a63f640 1 -- 172.21.15.123:0/1397054142 >> 172.21.15.123:0/1397054142 conn(0x7f3344070e10 msgr2=0x7f33440f9a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:54:20.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:20.459+0000 7f334a63f640 1 -- 172.21.15.123:0/1397054142 shutdown_connections 2024-06-18T02:54:20.460 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:20.459+0000 7f334a63f640 1 -- 172.21.15.123:0/1397054142 wait complete. 2024-06-18T02:54:20.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:20 smithi016 ceph-mon[29214]: from='client.? 172.21.15.123:0/1397054142' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-18T02:54:22.033 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-18T02:54:22.034 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph mon dump -f json 2024-06-18T02:54:22.304 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-18T02:54:22.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:22 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:54:22.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:22 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:54:22.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:22 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:54:22.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:22 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:54:22.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:22 smithi016 ceph-mon[29214]: Deploying daemon node-exporter.smithi123 on smithi123 2024-06-18T02:54:24.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:24.331+0000 7f07849f0640 1 -- 172.21.15.123:0/2299453603 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0780100e40 msgr2=0x7f0780101210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:54:24.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:24.331+0000 7f07849f0640 1 --2- 172.21.15.123:0/2299453603 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0780100e40 0x7f0780101210 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f0770009bd0 tx=0x7f0770004d10 comp rx=0 tx=0).stop 2024-06-18T02:54:24.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:24.331+0000 7f07849f0640 1 -- 172.21.15.123:0/2299453603 shutdown_connections 2024-06-18T02:54:24.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:24.331+0000 7f07849f0640 1 --2- 172.21.15.123:0/2299453603 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0780100e40 0x7f0780101210 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:54:24.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:24.331+0000 7f07849f0640 1 -- 172.21.15.123:0/2299453603 >> 172.21.15.123:0/2299453603 conn(0x7f0780070e10 msgr2=0x7f0780071210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:54:24.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:24.331+0000 7f07849f0640 1 -- 172.21.15.123:0/2299453603 shutdown_connections 2024-06-18T02:54:24.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:24.331+0000 7f07849f0640 1 -- 172.21.15.123:0/2299453603 wait complete. 2024-06-18T02:54:24.406 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:24.332+0000 7f07849f0640 1 Processor -- start 2024-06-18T02:54:24.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:24.332+0000 7f07849f0640 1 -- start start 2024-06-18T02:54:24.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:24.332+0000 7f07849f0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0780100e40 0x7f0780112620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:54:24.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:24.332+0000 7f07849f0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0780112b60 con 0x7f0780100e40 2024-06-18T02:54:24.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:24.332+0000 7f077e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0780100e40 0x7f0780112620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:54:24.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:24.332+0000 7f077e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0780100e40 0x7f0780112620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.123:52852/0 (socket says 172.21.15.123:52852) 2024-06-18T02:54:24.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:24.332+0000 7f077e59c640 1 -- 172.21.15.123:0/2648416842 learned_addr learned my addr 172.21.15.123:0/2648416842 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-06-18T02:54:24.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:24.333+0000 7f077e59c640 1 -- 172.21.15.123:0/2648416842 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07700098b0 con 0x7f0780100e40 2024-06-18T02:54:24.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:24.333+0000 7f077e59c640 1 --2- 172.21.15.123:0/2648416842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0780100e40 0x7f0780112620 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f0770009ba0 tx=0x7f0770004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:54:24.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:24.333+0000 7f0776ffd640 1 -- 172.21.15.123:0/2648416842 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f07700328b0 con 0x7f0780100e40 2024-06-18T02:54:24.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:24.333+0000 7f07849f0640 1 -- 172.21.15.123:0/2648416842 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0780112d60 con 0x7f0780100e40 2024-06-18T02:54:24.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:24.333+0000 7f07849f0640 1 -- 172.21.15.123:0/2648416842 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07801131b0 con 0x7f0780100e40 2024-06-18T02:54:24.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:24.333+0000 7f0776ffd640 1 -- 172.21.15.123:0/2648416842 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f077003d420 con 0x7f0780100e40 2024-06-18T02:54:24.407 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:24.333+0000 7f0776ffd640 1 -- 172.21.15.123:0/2648416842 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f07700465c0 con 0x7f0780100e40 2024-06-18T02:54:24.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:24.334+0000 7f07849f0640 1 -- 172.21.15.123:0/2648416842 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07801122d0 con 0x7f0780100e40 2024-06-18T02:54:24.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:24.334+0000 7f0776ffd640 1 -- 172.21.15.123:0/2648416842 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) v1 ==== 42059+0+0 (secure 0 0 0) 0x7f077002f030 con 0x7f0780100e40 2024-06-18T02:54:24.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:24.334+0000 7f0776ffd640 1 --2- 172.21.15.123:0/2648416842 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f076c0349f0 0x7f076c036ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:54:24.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:24.334+0000 7f0776ffd640 1 -- 172.21.15.123:0/2648416842 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f0770067c40 con 0x7f0780100e40 2024-06-18T02:54:24.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:24.335+0000 7f077dd9b640 1 --2- 172.21.15.123:0/2648416842 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f076c0349f0 0x7f076c036ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:54:24.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:24.335+0000 7f077dd9b640 1 --2- 172.21.15.123:0/2648416842 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f076c0349f0 0x7f076c036ea0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0768006fd0 tx=0x7f0768006db0 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:54:24.408 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:24.339+0000 7f0776ffd640 1 -- 172.21.15.123:0/2648416842 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f077004e940 con 0x7f0780100e40 2024-06-18T02:54:24.577 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:24.577+0000 7f07849f0640 1 -- 172.21.15.123:0/2648416842 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f07800fc5b0 con 0x7f0780100e40 2024-06-18T02:54:24.578 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:24.578+0000 7f0776ffd640 1 -- 172.21.15.123:0/2648416842 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f0770041030 con 0x7f0780100e40 2024-06-18T02:54:24.578 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:54:24.579 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"9e93709a-2d1d-11ef-bc9d-c7b262605968","modified":"2024-06-18T02:51:51.645370Z","created":"2024-06-18T02:51:51.645370Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi016","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-18T02:54:24.579 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-06-18T02:54:24.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:24.580+0000 7f07849f0640 1 -- 172.21.15.123:0/2648416842 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f076c0349f0 msgr2=0x7f076c036ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:54:24.580 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:24.580+0000 7f07849f0640 1 --2- 172.21.15.123:0/2648416842 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f076c0349f0 0x7f076c036ea0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0768006fd0 tx=0x7f0768006db0 comp rx=0 tx=0).stop 2024-06-18T02:54:24.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:24.580+0000 7f07849f0640 1 -- 172.21.15.123:0/2648416842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0780100e40 msgr2=0x7f0780112620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:54:24.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:24.580+0000 7f07849f0640 1 --2- 172.21.15.123:0/2648416842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0780100e40 0x7f0780112620 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f0770009ba0 tx=0x7f0770004d10 comp rx=0 tx=0).stop 2024-06-18T02:54:24.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:24.580+0000 7f07849f0640 1 -- 172.21.15.123:0/2648416842 shutdown_connections 2024-06-18T02:54:24.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:24.580+0000 7f07849f0640 1 --2- 172.21.15.123:0/2648416842 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f076c0349f0 0x7f076c036ea0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:54:24.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:24.580+0000 7f07849f0640 1 --2- 172.21.15.123:0/2648416842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0780100e40 0x7f0780112620 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:54:24.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:24.580+0000 7f07849f0640 1 -- 172.21.15.123:0/2648416842 >> 172.21.15.123:0/2648416842 conn(0x7f0780070e10 msgr2=0x7f07800646f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:54:24.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:24.580+0000 7f07849f0640 1 -- 172.21.15.123:0/2648416842 shutdown_connections 2024-06-18T02:54:24.581 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:24.580+0000 7f07849f0640 1 -- 172.21.15.123:0/2648416842 wait complete. 2024-06-18T02:54:25.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:24 smithi016 ceph-mon[29214]: from='client.? 172.21.15.123:0/2648416842' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-18T02:54:26.301 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-18T02:54:26.301 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph mon dump -f json 2024-06-18T02:54:26.626 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-18T02:54:27.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:27 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:54:27.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:27 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:54:28.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:28.816+0000 7f627a136640 1 -- 172.21.15.123:0/3519843226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6274101f40 msgr2=0x7f6274102310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:54:28.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:28.816+0000 7f627a136640 1 --2- 172.21.15.123:0/3519843226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6274101f40 0x7f6274102310 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6264004850 tx=0x7f6264009da0 comp rx=0 tx=0).stop 2024-06-18T02:54:28.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:28.816+0000 7f627a136640 1 -- 172.21.15.123:0/3519843226 shutdown_connections 2024-06-18T02:54:28.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:28.816+0000 7f627a136640 1 --2- 172.21.15.123:0/3519843226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6274101f40 0x7f6274102310 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:54:28.822 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:28.816+0000 7f627a136640 1 -- 172.21.15.123:0/3519843226 >> 172.21.15.123:0/3519843226 conn(0x7f62740f7af0 msgr2=0x7f62740f9f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:54:28.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:28.816+0000 7f627a136640 1 -- 172.21.15.123:0/3519843226 shutdown_connections 2024-06-18T02:54:28.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:28.816+0000 7f627a136640 1 -- 172.21.15.123:0/3519843226 wait complete. 2024-06-18T02:54:28.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:28.816+0000 7f627a136640 1 Processor -- start 2024-06-18T02:54:28.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:28.816+0000 7f627a136640 1 -- start start 2024-06-18T02:54:28.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:28.816+0000 7f627a136640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6274101f40 0x7f62741147b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:54:28.823 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:28.817+0000 7f627a136640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6274114cf0 con 0x7f6274101f40 2024-06-18T02:54:28.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:28.817+0000 7f62737fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6274101f40 0x7f62741147b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:54:28.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:28.817+0000 7f62737fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6274101f40 0x7f62741147b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.123:52856/0 (socket says 172.21.15.123:52856) 2024-06-18T02:54:28.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:28.817+0000 7f62737fe640 1 -- 172.21.15.123:0/2891639376 learned_addr learned my addr 172.21.15.123:0/2891639376 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-06-18T02:54:28.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:28.817+0000 7f62737fe640 1 -- 172.21.15.123:0/2891639376 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6264005040 con 0x7f6274101f40 2024-06-18T02:54:28.824 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:28.817+0000 7f62737fe640 1 --2- 172.21.15.123:0/2891639376 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6274101f40 0x7f62741147b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f6264004820 tx=0x7f6264009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:54:28.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:28.818+0000 7f6270ff9640 1 -- 172.21.15.123:0/2891639376 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f626402cf10 con 0x7f6274101f40 2024-06-18T02:54:28.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:28.818+0000 7f627a136640 1 -- 172.21.15.123:0/2891639376 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6274114ef0 con 0x7f6274101f40 2024-06-18T02:54:28.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:28.818+0000 7f6270ff9640 1 -- 172.21.15.123:0/2891639376 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6264033b00 con 0x7f6274101f40 2024-06-18T02:54:28.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:28.818+0000 7f627a136640 1 -- 172.21.15.123:0/2891639376 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6274115360 con 0x7f6274101f40 2024-06-18T02:54:28.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:28.818+0000 7f6270ff9640 1 -- 172.21.15.123:0/2891639376 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f626403db70 con 0x7f6274101f40 2024-06-18T02:54:28.825 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:28.819+0000 7f6270ff9640 1 -- 172.21.15.123:0/2891639376 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) v1 ==== 42059+0+0 (secure 0 0 0) 0x7f6264030020 con 0x7f6274101f40 2024-06-18T02:54:28.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:28.819+0000 7f627a136640 1 -- 172.21.15.123:0/2891639376 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62740fd6b0 con 0x7f6274101f40 2024-06-18T02:54:28.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:28.819+0000 7f6270ff9640 1 --2- 172.21.15.123:0/2891639376 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f62600349f0 0x7f6260036ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:54:28.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:28.819+0000 7f6270ff9640 1 -- 172.21.15.123:0/2891639376 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f62640670a0 con 0x7f6274101f40 2024-06-18T02:54:28.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:28.819+0000 7f626bfff640 1 --2- 172.21.15.123:0/2891639376 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f62600349f0 0x7f6260036ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:54:28.826 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:28.819+0000 7f626bfff640 1 --2- 172.21.15.123:0/2891639376 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f62600349f0 0x7f6260036ea0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f625c006fd0 tx=0x7f625c006db0 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:54:28.827 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:28.824+0000 7f6270ff9640 1 -- 172.21.15.123:0/2891639376 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f6264045570 con 0x7f6274101f40 2024-06-18T02:54:29.064 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:29.064+0000 7f627a136640 1 -- 172.21.15.123:0/2891639376 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f627405eb20 con 0x7f6274101f40 2024-06-18T02:54:29.065 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:29.064+0000 7f6270ff9640 1 -- 172.21.15.123:0/2891639376 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f6264045390 con 0x7f6274101f40 2024-06-18T02:54:29.065 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:54:29.065 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"9e93709a-2d1d-11ef-bc9d-c7b262605968","modified":"2024-06-18T02:51:51.645370Z","created":"2024-06-18T02:51:51.645370Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi016","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-18T02:54:29.065 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-06-18T02:54:29.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:29.066+0000 7f627a136640 1 -- 172.21.15.123:0/2891639376 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f62600349f0 msgr2=0x7f6260036ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:54:29.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:29.066+0000 7f627a136640 1 --2- 172.21.15.123:0/2891639376 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f62600349f0 0x7f6260036ea0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f625c006fd0 tx=0x7f625c006db0 comp rx=0 tx=0).stop 2024-06-18T02:54:29.066 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:29.066+0000 7f627a136640 1 -- 172.21.15.123:0/2891639376 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6274101f40 msgr2=0x7f62741147b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:54:29.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:29.066+0000 7f627a136640 1 --2- 172.21.15.123:0/2891639376 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6274101f40 0x7f62741147b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f6264004820 tx=0x7f6264009da0 comp rx=0 tx=0).stop 2024-06-18T02:54:29.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:29.066+0000 7f627a136640 1 -- 172.21.15.123:0/2891639376 shutdown_connections 2024-06-18T02:54:29.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:29.066+0000 7f627a136640 1 --2- 172.21.15.123:0/2891639376 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f62600349f0 0x7f6260036ea0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:54:29.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:29.066+0000 7f627a136640 1 --2- 172.21.15.123:0/2891639376 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6274101f40 0x7f62741147b0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:54:29.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:29.066+0000 7f627a136640 1 -- 172.21.15.123:0/2891639376 >> 172.21.15.123:0/2891639376 conn(0x7f62740f7af0 msgr2=0x7f62740f8750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:54:29.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:29.066+0000 7f627a136640 1 -- 172.21.15.123:0/2891639376 shutdown_connections 2024-06-18T02:54:29.067 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:29.066+0000 7f627a136640 1 -- 172.21.15.123:0/2891639376 wait complete. 2024-06-18T02:54:29.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:29 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:54:29.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:29 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:54:29.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:29 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:54:29.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:29 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:54:29.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:29 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi123.olpkyi", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-18T02:54:29.806 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:29 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi123.olpkyi", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-06-18T02:54:29.806 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:29 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-18T02:54:29.806 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:29 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:54:29.806 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:29 smithi016 ceph-mon[29214]: Deploying daemon mgr.smithi123.olpkyi on smithi123 2024-06-18T02:54:29.806 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:29 smithi016 ceph-mon[29214]: from='client.? 172.21.15.123:0/2891639376' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-18T02:54:30.686 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-18T02:54:30.686 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph mon dump -f json 2024-06-18T02:54:31.529 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-18T02:54:32.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:32 smithi016 ceph-mon[29214]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:54:32.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:32 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:54:32.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:32 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:54:32.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:32 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:54:32.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:32 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:54:32.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:32 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:54:32.556 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:32 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-18T02:54:32.556 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:32 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:54:33.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:33.250+0000 7fab3283a640 1 -- 172.21.15.123:0/3496224356 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab2c0fcf70 msgr2=0x7fab2c0fd340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:54:33.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:33.250+0000 7fab3283a640 1 --2- 172.21.15.123:0/3496224356 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab2c0fcf70 0x7fab2c0fd340 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fab1c004850 tx=0x7fab1c009da0 comp rx=0 tx=0).stop 2024-06-18T02:54:33.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:33.250+0000 7fab3283a640 1 -- 172.21.15.123:0/3496224356 shutdown_connections 2024-06-18T02:54:33.254 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:33.250+0000 7fab3283a640 1 --2- 172.21.15.123:0/3496224356 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab2c0fcf70 0x7fab2c0fd340 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:54:33.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:33.250+0000 7fab3283a640 1 -- 172.21.15.123:0/3496224356 >> 172.21.15.123:0/3496224356 conn(0x7fab2c0f6a70 msgr2=0x7fab2c0f8e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:54:33.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:33.250+0000 7fab3283a640 1 -- 172.21.15.123:0/3496224356 shutdown_connections 2024-06-18T02:54:33.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:33.250+0000 7fab3283a640 1 -- 172.21.15.123:0/3496224356 wait complete. 2024-06-18T02:54:33.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:33.250+0000 7fab3283a640 1 Processor -- start 2024-06-18T02:54:33.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:33.251+0000 7fab3283a640 1 -- start start 2024-06-18T02:54:33.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:33.251+0000 7fab3283a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab2c0fcf70 0x7fab2c112690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:54:33.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:33.251+0000 7fab3283a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab2c112bd0 con 0x7fab2c0fcf70 2024-06-18T02:54:33.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:33.251+0000 7fab2bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab2c0fcf70 0x7fab2c112690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:54:33.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:33.251+0000 7fab2bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab2c0fcf70 0x7fab2c112690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.123:38564/0 (socket says 172.21.15.123:38564) 2024-06-18T02:54:33.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:33.251+0000 7fab2bfff640 1 -- 172.21.15.123:0/1438479057 learned_addr learned my addr 172.21.15.123:0/1438479057 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-06-18T02:54:33.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:33.251+0000 7fab2bfff640 1 -- 172.21.15.123:0/1438479057 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab1c005040 con 0x7fab2c0fcf70 2024-06-18T02:54:33.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:33.252+0000 7fab2bfff640 1 --2- 172.21.15.123:0/1438479057 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab2c0fcf70 0x7fab2c112690 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fab1c004820 tx=0x7fab1c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:54:33.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:33.252+0000 7fab28ff9640 1 -- 172.21.15.123:0/1438479057 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab1c02cf10 con 0x7fab2c0fcf70 2024-06-18T02:54:33.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:33.252+0000 7fab3283a640 1 -- 172.21.15.123:0/1438479057 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab2c112dd0 con 0x7fab2c0fcf70 2024-06-18T02:54:33.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:33.252+0000 7fab28ff9640 1 -- 172.21.15.123:0/1438479057 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fab1c033b00 con 0x7fab2c0fcf70 2024-06-18T02:54:33.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:33.252+0000 7fab28ff9640 1 -- 172.21.15.123:0/1438479057 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab1c03db70 con 0x7fab2c0fcf70 2024-06-18T02:54:33.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:33.252+0000 7fab3283a640 1 -- 172.21.15.123:0/1438479057 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab2c1131c0 con 0x7fab2c0fcf70 2024-06-18T02:54:33.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:33.253+0000 7fab28ff9640 1 -- 172.21.15.123:0/1438479057 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 17) v1 ==== 42059+0+0 (secure 0 0 0) 0x7fab1c030020 con 0x7fab2c0fcf70 2024-06-18T02:54:33.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:33.253+0000 7fab3283a640 1 -- 172.21.15.123:0/1438479057 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab2c0657e0 con 0x7fab2c0fcf70 2024-06-18T02:54:33.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:33.253+0000 7fab28ff9640 1 --2- 172.21.15.123:0/1438479057 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fab180349f0 0x7fab18036ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:54:33.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:33.253+0000 7fab28ff9640 1 -- 172.21.15.123:0/1438479057 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fab1c066df0 con 0x7fab2c0fcf70 2024-06-18T02:54:33.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:33.253+0000 7fab2b7fe640 1 --2- 172.21.15.123:0/1438479057 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fab180349f0 0x7fab18036ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:54:33.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:33.254+0000 7fab2b7fe640 1 --2- 172.21.15.123:0/1438479057 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fab180349f0 0x7fab18036ea0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fab14006fd0 tx=0x7fab14006db0 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:54:33.259 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:33.258+0000 7fab28ff9640 1 -- 172.21.15.123:0/1438479057 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fab1c03ddf0 con 0x7fab2c0fcf70 2024-06-18T02:54:33.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:33.501+0000 7fab3283a640 1 -- 172.21.15.123:0/1438479057 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fab2c04b480 con 0x7fab2c0fcf70 2024-06-18T02:54:33.502 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:33.501+0000 7fab28ff9640 1 -- 172.21.15.123:0/1438479057 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fab1c033c70 con 0x7fab2c0fcf70 2024-06-18T02:54:33.503 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:54:33.503 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"9e93709a-2d1d-11ef-bc9d-c7b262605968","modified":"2024-06-18T02:51:51.645370Z","created":"2024-06-18T02:51:51.645370Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi016","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-18T02:54:33.503 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-06-18T02:54:33.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:33.503+0000 7fab3283a640 1 -- 172.21.15.123:0/1438479057 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fab180349f0 msgr2=0x7fab18036ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:54:33.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:33.503+0000 7fab3283a640 1 --2- 172.21.15.123:0/1438479057 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fab180349f0 0x7fab18036ea0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fab14006fd0 tx=0x7fab14006db0 comp rx=0 tx=0).stop 2024-06-18T02:54:33.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:33.503+0000 7fab3283a640 1 -- 172.21.15.123:0/1438479057 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab2c0fcf70 msgr2=0x7fab2c112690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:54:33.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:33.503+0000 7fab3283a640 1 --2- 172.21.15.123:0/1438479057 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab2c0fcf70 0x7fab2c112690 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fab1c004820 tx=0x7fab1c009da0 comp rx=0 tx=0).stop 2024-06-18T02:54:33.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:33.504+0000 7fab3283a640 1 -- 172.21.15.123:0/1438479057 shutdown_connections 2024-06-18T02:54:33.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:33.504+0000 7fab3283a640 1 --2- 172.21.15.123:0/1438479057 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fab180349f0 0x7fab18036ea0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:54:33.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:33.504+0000 7fab3283a640 1 --2- 172.21.15.123:0/1438479057 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab2c0fcf70 0x7fab2c112690 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:54:33.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:33.504+0000 7fab3283a640 1 -- 172.21.15.123:0/1438479057 >> 172.21.15.123:0/1438479057 conn(0x7fab2c0f6a70 msgr2=0x7fab2c102c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:54:33.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:33.504+0000 7fab3283a640 1 -- 172.21.15.123:0/1438479057 shutdown_connections 2024-06-18T02:54:33.505 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:33.504+0000 7fab3283a640 1 -- 172.21.15.123:0/1438479057 wait complete. 2024-06-18T02:54:33.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:33 smithi016 ceph-mon[29214]: Deploying daemon mon.smithi123 on smithi123 2024-06-18T02:54:34.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:34 smithi016 ceph-mon[29214]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:54:34.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:34 smithi016 ceph-mon[29214]: from='client.? 172.21.15.123:0/1438479057' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-18T02:54:35.394 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-18T02:54:35.394 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph mon dump -f json 2024-06-18T02:54:35.841 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi123/config 2024-06-18T02:54:36.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:36 smithi016 ceph-mon[29214]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:54:36.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:36 smithi016 ceph-mon[29214]: Standby manager daemon smithi123.olpkyi started 2024-06-18T02:54:36.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:36 smithi016 ceph-mon[29214]: from='mgr.? 172.21.15.123:0/1907727345' entity='mgr.smithi123.olpkyi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi123.olpkyi/crt"}]: dispatch 2024-06-18T02:54:36.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:36 smithi016 ceph-mon[29214]: from='mgr.? 172.21.15.123:0/1907727345' entity='mgr.smithi123.olpkyi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-18T02:54:36.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:36 smithi016 ceph-mon[29214]: from='mgr.? 172.21.15.123:0/1907727345' entity='mgr.smithi123.olpkyi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi123.olpkyi/key"}]: dispatch 2024-06-18T02:54:36.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:36 smithi016 ceph-mon[29214]: from='mgr.? 172.21.15.123:0/1907727345' entity='mgr.smithi123.olpkyi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-18T02:54:37.427 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:37.422+0000 7fb4a8a35640 1 -- 172.21.15.123:0/2639710910 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4a4100fd0 msgr2=0x7fb4a41013a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:54:37.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:37.422+0000 7fb4a8a35640 1 --2- 172.21.15.123:0/2639710910 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4a4100fd0 0x7fb4a41013a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fb494004850 tx=0x7fb494009da0 comp rx=0 tx=0).stop 2024-06-18T02:54:37.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:37.422+0000 7fb4a8a35640 1 -- 172.21.15.123:0/2639710910 shutdown_connections 2024-06-18T02:54:37.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:37.422+0000 7fb4a8a35640 1 --2- 172.21.15.123:0/2639710910 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4a4100fd0 0x7fb4a41013a0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:54:37.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:37.422+0000 7fb4a8a35640 1 -- 172.21.15.123:0/2639710910 >> 172.21.15.123:0/2639710910 conn(0x7fb4a40f6ba0 msgr2=0x7fb4a40f8fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:54:37.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:37.422+0000 7fb4a8a35640 1 -- 172.21.15.123:0/2639710910 shutdown_connections 2024-06-18T02:54:37.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:37.422+0000 7fb4a8a35640 1 -- 172.21.15.123:0/2639710910 wait complete. 2024-06-18T02:54:37.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:37.423+0000 7fb4a8a35640 1 Processor -- start 2024-06-18T02:54:37.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:37.423+0000 7fb4a8a35640 1 -- start start 2024-06-18T02:54:37.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:37.423+0000 7fb4a8a35640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4a4100fd0 0x7fb4a4112780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:54:37.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:37.423+0000 7fb4a8a35640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a4112cc0 con 0x7fb4a4100fd0 2024-06-18T02:54:37.428 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:37.424+0000 7fb4a259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4a4100fd0 0x7fb4a4112780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:54:37.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:37.424+0000 7fb4a259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4a4100fd0 0x7fb4a4112780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.123:38582/0 (socket says 172.21.15.123:38582) 2024-06-18T02:54:37.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:37.424+0000 7fb4a259c640 1 -- 172.21.15.123:0/2354159922 learned_addr learned my addr 172.21.15.123:0/2354159922 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-06-18T02:54:37.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:37.424+0000 7fb4a259c640 1 -- 172.21.15.123:0/2354159922 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb494005040 con 0x7fb4a4100fd0 2024-06-18T02:54:37.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:37.424+0000 7fb4a259c640 1 --2- 172.21.15.123:0/2354159922 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4a4100fd0 0x7fb4a4112780 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb494004820 tx=0x7fb494009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:54:37.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:37.424+0000 7fb49affd640 1 -- 172.21.15.123:0/2354159922 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb49402cf10 con 0x7fb4a4100fd0 2024-06-18T02:54:37.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:37.424+0000 7fb4a8a35640 1 -- 172.21.15.123:0/2354159922 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4a4112ec0 con 0x7fb4a4100fd0 2024-06-18T02:54:37.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:37.425+0000 7fb49affd640 1 -- 172.21.15.123:0/2354159922 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb494033b00 con 0x7fb4a4100fd0 2024-06-18T02:54:37.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:37.425+0000 7fb49affd640 1 -- 172.21.15.123:0/2354159922 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb49403db70 con 0x7fb4a4100fd0 2024-06-18T02:54:37.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:37.425+0000 7fb4a8a35640 1 -- 172.21.15.123:0/2354159922 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4a41132b0 con 0x7fb4a4100fd0 2024-06-18T02:54:37.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:37.426+0000 7fb4a8a35640 1 -- 172.21.15.123:0/2354159922 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4a40fc740 con 0x7fb4a4100fd0 2024-06-18T02:54:37.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:37.426+0000 7fb49affd640 1 -- 172.21.15.123:0/2354159922 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) v1 ==== 83689+0+0 (secure 0 0 0) 0x7fb494030020 con 0x7fb4a4100fd0 2024-06-18T02:54:37.429 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:37.427+0000 7fb49affd640 1 --2- 172.21.15.123:0/2354159922 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fb490065370 0x7fb490067820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:54:37.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:37.427+0000 7fb49affd640 1 -- 172.21.15.123:0/2354159922 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fb4940a2260 con 0x7fb4a4100fd0 2024-06-18T02:54:37.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:37.427+0000 7fb49bfff640 1 --2- 172.21.15.123:0/2354159922 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fb490065370 0x7fb490067820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:54:37.430 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:37.427+0000 7fb49bfff640 1 --2- 172.21.15.123:0/2354159922 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fb490065370 0x7fb490067820 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fb48c006fd0 tx=0x7fb48c006db0 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:54:37.432 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:37.431+0000 7fb49affd640 1 -- 172.21.15.123:0/2354159922 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fb494071510 con 0x7fb4a4100fd0 2024-06-18T02:54:37.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:37 smithi016 ceph-mon[29214]: Failed to load ceph-mgr modules: k8sevents 2024-06-18T02:54:37.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:37 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:54:37.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:37 smithi016 ceph-mon[29214]: mgrmap e18: smithi016.amxydz(active, since 25s), standbys: smithi123.olpkyi 2024-06-18T02:54:37.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:37 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "mgr metadata", "who": "smithi123.olpkyi", "id": "smithi123.olpkyi"}]: dispatch 2024-06-18T02:54:37.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:37.662+0000 7fb4a8a35640 1 -- 172.21.15.123:0/2354159922 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb4a4105380 con 0x7fb4a4100fd0 2024-06-18T02:54:37.663 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:37.662+0000 7fb49affd640 1 -- 172.21.15.123:0/2354159922 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fb49406fe60 con 0x7fb4a4100fd0 2024-06-18T02:54:37.664 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:54:37.664 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":1,"fsid":"9e93709a-2d1d-11ef-bc9d-c7b262605968","modified":"2024-06-18T02:51:51.645370Z","created":"2024-06-18T02:51:51.645370Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi016","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-18T02:54:37.664 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 1 2024-06-18T02:54:37.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:37.665+0000 7fb4a8a35640 1 -- 172.21.15.123:0/2354159922 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fb490065370 msgr2=0x7fb490067820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:54:37.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:37.665+0000 7fb4a8a35640 1 --2- 172.21.15.123:0/2354159922 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fb490065370 0x7fb490067820 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fb48c006fd0 tx=0x7fb48c006db0 comp rx=0 tx=0).stop 2024-06-18T02:54:37.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:37.665+0000 7fb4a8a35640 1 -- 172.21.15.123:0/2354159922 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4a4100fd0 msgr2=0x7fb4a4112780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:54:37.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:37.665+0000 7fb4a8a35640 1 --2- 172.21.15.123:0/2354159922 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4a4100fd0 0x7fb4a4112780 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb494004820 tx=0x7fb494009da0 comp rx=0 tx=0).stop 2024-06-18T02:54:37.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:37.665+0000 7fb4a8a35640 1 -- 172.21.15.123:0/2354159922 shutdown_connections 2024-06-18T02:54:37.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:37.665+0000 7fb4a8a35640 1 --2- 172.21.15.123:0/2354159922 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fb490065370 0x7fb490067820 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:54:37.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:37.665+0000 7fb4a8a35640 1 --2- 172.21.15.123:0/2354159922 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4a4100fd0 0x7fb4a4112780 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:54:37.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:37.665+0000 7fb4a8a35640 1 -- 172.21.15.123:0/2354159922 >> 172.21.15.123:0/2354159922 conn(0x7fb4a40f6ba0 msgr2=0x7fb4a40654e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:54:37.666 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:37.665+0000 7fb4a8a35640 1 -- 172.21.15.123:0/2354159922 shutdown_connections 2024-06-18T02:54:37.667 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:37.665+0000 7fb4a8a35640 1 -- 172.21.15.123:0/2354159922 wait complete. 2024-06-18T02:54:39.600 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-18T02:54:39.601 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph mon dump -f json 2024-06-18T02:54:39.850 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi123/config 2024-06-18T02:54:43.228 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:42 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "mon metadata", "id": "smithi016"}]: dispatch 2024-06-18T02:54:43.229 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:42 smithi016 ceph-mon[29214]: mon.smithi016 calling monitor election 2024-06-18T02:54:43.229 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:42 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "mon metadata", "id": "smithi123"}]: dispatch 2024-06-18T02:54:43.229 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:42 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "mon metadata", "id": "smithi123"}]: dispatch 2024-06-18T02:54:43.229 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:42 smithi016 ceph-mon[29214]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:54:43.229 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:42 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "mon metadata", "id": "smithi123"}]: dispatch 2024-06-18T02:54:43.229 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:42 smithi016 ceph-mon[29214]: mon.smithi123 calling monitor election 2024-06-18T02:54:43.229 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:42 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "mon metadata", "id": "smithi123"}]: dispatch 2024-06-18T02:54:43.229 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:42 smithi016 ceph-mon[29214]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:54:43.229 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:42 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:54:43.229 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:42 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "mon metadata", "id": "smithi123"}]: dispatch 2024-06-18T02:54:43.229 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:42 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "mon metadata", "id": "smithi123"}]: dispatch 2024-06-18T02:54:43.229 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:42 smithi016 ceph-mon[29214]: mon.smithi016 is new leader, mons smithi016,smithi123 in quorum (ranks 0,1) 2024-06-18T02:54:43.229 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:42 smithi016 ceph-mon[29214]: monmap e2: 2 mons at {smithi016=[v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0],smithi123=[v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-06-18T02:54:43.230 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:42 smithi016 ceph-mon[29214]: fsmap 2024-06-18T02:54:43.230 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:42 smithi016 ceph-mon[29214]: osdmap e5: 0 total, 0 up, 0 in 2024-06-18T02:54:43.230 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:42 smithi016 ceph-mon[29214]: mgrmap e18: smithi016.amxydz(active, since 31s), standbys: smithi123.olpkyi 2024-06-18T02:54:43.230 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:42 smithi016 ceph-mon[29214]: Health detail: HEALTH_WARN 2 mgr modules have recently crashed 2024-06-18T02:54:43.230 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:42 smithi016 ceph-mon[29214]: [WRN] RECENT_MGR_MODULE_CRASH: 2 mgr modules have recently crashed 2024-06-18T02:54:43.230 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:42 smithi016 ceph-mon[29214]: mgr module k8sevents crashed in daemon mgr.smithi016.amxydz on host smithi016 at 2024-06-18T02:52:13.841616Z 2024-06-18T02:54:43.230 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:42 smithi016 ceph-mon[29214]: mgr module k8sevents crashed in daemon mgr.smithi016.amxydz on host smithi016 at 2024-06-18T02:52:23.627404Z 2024-06-18T02:54:43.230 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:42 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:54:43.230 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:42 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:54:43.230 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:42 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:54:43.230 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:42 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:54:43.230 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:42 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:54:43.234 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:54:42 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:54:44.204 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:43 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:54:44.204 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:43 smithi016 ceph-mon[29214]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:54:44.204 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:43 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "mon metadata", "id": "smithi123"}]: dispatch 2024-06-18T02:54:44.214 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:54:43 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:54:44.214 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:54:43 smithi123 ceph-mon[36358]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:54:44.214 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:54:43 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "mon metadata", "id": "smithi123"}]: dispatch 2024-06-18T02:54:44.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:44.352+0000 7fce0aea6640 1 -- 172.21.15.123:0/2601668390 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcdf40034f0 msgr2=0x7fcdf4005970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:54:44.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:44.352+0000 7fce0aea6640 1 --2- 172.21.15.123:0/2601668390 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcdf40034f0 0x7fcdf4005970 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fce040fbfe0 tx=0x7fcdf800b040 comp rx=0 tx=0).stop 2024-06-18T02:54:44.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:44.352+0000 7fce0aea6640 1 -- 172.21.15.123:0/2601668390 shutdown_connections 2024-06-18T02:54:44.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:44.352+0000 7fce0aea6640 1 --2- 172.21.15.123:0/2601668390 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcdf40034f0 0x7fcdf4005970 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:54:44.354 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:44.352+0000 7fce0aea6640 1 -- 172.21.15.123:0/2601668390 >> 172.21.15.123:0/2601668390 conn(0x7fce040f7bf0 msgr2=0x7fce040fa000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:54:44.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:44.352+0000 7fce0aea6640 1 -- 172.21.15.123:0/2601668390 shutdown_connections 2024-06-18T02:54:44.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:44.352+0000 7fce0aea6640 1 -- 172.21.15.123:0/2601668390 wait complete. 2024-06-18T02:54:44.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:44.353+0000 7fce0aea6640 1 Processor -- start 2024-06-18T02:54:44.355 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:44.353+0000 7fce0aea6640 1 -- start start 2024-06-18T02:54:44.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:44.354+0000 7fce0aea6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcdf40034f0 0x7fce0406f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:54:44.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:44.354+0000 7fce0aea6640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce0406fcb0 0x7fce04070100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:54:44.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:44.354+0000 7fce0aea6640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce04118af0 con 0x7fcdf40034f0 2024-06-18T02:54:44.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:44.354+0000 7fce0aea6640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce04118c30 con 0x7fce0406fcb0 2024-06-18T02:54:44.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:44.354+0000 7fce03fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce0406fcb0 0x7fce04070100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:54:44.357 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:44.354+0000 7fce03fff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce0406fcb0 0x7fce04070100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37096/0 (socket says 172.21.15.123:37096) 2024-06-18T02:54:44.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:44.354+0000 7fce03fff640 1 -- 172.21.15.123:0/7128576 learned_addr learned my addr 172.21.15.123:0/7128576 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-06-18T02:54:44.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:44.354+0000 7fce08c42640 1 --2- 172.21.15.123:0/7128576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcdf40034f0 0x7fce0406f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:54:44.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:44.354+0000 7fce03fff640 1 -- 172.21.15.123:0/7128576 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce0406fcb0 msgr2=0x7fce04070100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-06-18T02:54:44.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:44.354+0000 7fce03fff640 1 -- 172.21.15.123:0/7128576 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce0406fcb0 msgr2=0x7fce04070100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-06-18T02:54:44.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:44.354+0000 7fce03fff640 1 --2- 172.21.15.123:0/7128576 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce0406fcb0 0x7fce04070100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-06-18T02:54:44.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:44.354+0000 7fce03fff640 1 --2- 172.21.15.123:0/7128576 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce0406fcb0 0x7fce04070100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-18T02:54:44.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:44.354+0000 7fce08c42640 1 -- 172.21.15.123:0/7128576 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce0406fcb0 msgr2=0x7fce04070100 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-18T02:54:44.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:44.354+0000 7fce08c42640 1 --2- 172.21.15.123:0/7128576 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce0406fcb0 0x7fce04070100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:54:44.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:44.354+0000 7fce08c42640 1 -- 172.21.15.123:0/7128576 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdf8005040 con 0x7fcdf40034f0 2024-06-18T02:54:44.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:44.354+0000 7fce08c42640 1 --2- 172.21.15.123:0/7128576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcdf40034f0 0x7fce0406f770 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fcdf8009c50 tx=0x7fcdf800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:54:44.358 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:44.355+0000 7fce017fa640 1 -- 172.21.15.123:0/7128576 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcdf8030cf0 con 0x7fcdf40034f0 2024-06-18T02:54:44.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:44.355+0000 7fce017fa640 1 -- 172.21.15.123:0/7128576 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcdf800f040 con 0x7fcdf40034f0 2024-06-18T02:54:44.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:44.355+0000 7fce0aea6640 1 -- 172.21.15.123:0/7128576 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce04118dd0 con 0x7fcdf40034f0 2024-06-18T02:54:44.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:44.355+0000 7fce017fa640 1 -- 172.21.15.123:0/7128576 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcdf803c650 con 0x7fcdf40034f0 2024-06-18T02:54:44.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:44.355+0000 7fce0aea6640 1 -- 172.21.15.123:0/7128576 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce04119290 con 0x7fcdf40034f0 2024-06-18T02:54:44.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:44.356+0000 7fce0aea6640 1 -- 172.21.15.123:0/7128576 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce040fd7f0 con 0x7fcdf40034f0 2024-06-18T02:54:44.359 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:44.356+0000 7fce017fa640 1 -- 172.21.15.123:0/7128576 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) v1 ==== 83689+0+0 (secure 0 0 0) 0x7fcdf803e440 con 0x7fcdf40034f0 2024-06-18T02:54:44.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:44.356+0000 7fce017fa640 1 --2- 172.21.15.123:0/7128576 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fcde80654b0 0x7fcde8067960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:54:44.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:44.357+0000 7fce017fa640 1 -- 172.21.15.123:0/7128576 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fcdf80a2980 con 0x7fcdf40034f0 2024-06-18T02:54:44.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:44.357+0000 7fce03fff640 1 --2- 172.21.15.123:0/7128576 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fcde80654b0 0x7fcde8067960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:54:44.360 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:44.357+0000 7fce03fff640 1 --2- 172.21.15.123:0/7128576 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fcde80654b0 0x7fcde8067960 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fcdf0006fd0 tx=0x7fcdf0008040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:54:44.361 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:44.361+0000 7fce017fa640 1 -- 172.21.15.123:0/7128576 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fcdf8071bd0 con 0x7fcdf40034f0 2024-06-18T02:54:44.591 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:44.590+0000 7fce0aea6640 1 -- 172.21.15.123:0/7128576 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fce0405ead0 con 0x7fcdf40034f0 2024-06-18T02:54:44.592 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:44.591+0000 7fce017fa640 1 -- 172.21.15.123:0/7128576 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1017 (secure 0 0 0) 0x7fcdf80a2130 con 0x7fcdf40034f0 2024-06-18T02:54:44.592 INFO:teuthology.orchestra.run.smithi123.stdout: 2024-06-18T02:54:44.592 INFO:teuthology.orchestra.run.smithi123.stdout:{"epoch":2,"fsid":"9e93709a-2d1d-11ef-bc9d-c7b262605968","modified":"2024-06-18T02:54:37.851976Z","created":"2024-06-18T02:51:51.645370Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi016","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:3300","nonce":0},{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-06-18T02:54:44.592 INFO:teuthology.orchestra.run.smithi123.stderr:dumped monmap epoch 2 2024-06-18T02:54:44.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:44.593+0000 7fce0aea6640 1 -- 172.21.15.123:0/7128576 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fcde80654b0 msgr2=0x7fcde8067960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:54:44.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:44.593+0000 7fce0aea6640 1 --2- 172.21.15.123:0/7128576 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fcde80654b0 0x7fcde8067960 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fcdf0006fd0 tx=0x7fcdf0008040 comp rx=0 tx=0).stop 2024-06-18T02:54:44.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:44.593+0000 7fce0aea6640 1 -- 172.21.15.123:0/7128576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcdf40034f0 msgr2=0x7fce0406f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:54:44.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:44.593+0000 7fce0aea6640 1 --2- 172.21.15.123:0/7128576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcdf40034f0 0x7fce0406f770 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fcdf8009c50 tx=0x7fcdf800b040 comp rx=0 tx=0).stop 2024-06-18T02:54:44.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:44.593+0000 7fce0aea6640 1 -- 172.21.15.123:0/7128576 shutdown_connections 2024-06-18T02:54:44.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:44.593+0000 7fce0aea6640 1 --2- 172.21.15.123:0/7128576 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fcde80654b0 0x7fcde8067960 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:54:44.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:44.593+0000 7fce0aea6640 1 --2- 172.21.15.123:0/7128576 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce0406fcb0 0x7fce04070100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:54:44.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:44.593+0000 7fce0aea6640 1 --2- 172.21.15.123:0/7128576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcdf40034f0 0x7fce0406f770 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:54:44.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:44.593+0000 7fce0aea6640 1 -- 172.21.15.123:0/7128576 >> 172.21.15.123:0/7128576 conn(0x7fce040f7bf0 msgr2=0x7fce041014e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:54:44.594 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:44.593+0000 7fce0aea6640 1 -- 172.21.15.123:0/7128576 shutdown_connections 2024-06-18T02:54:44.595 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:54:44.593+0000 7fce0aea6640 1 -- 172.21.15.123:0/7128576 wait complete. 2024-06-18T02:54:45.194 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:54:44 smithi123 ceph-mon[36358]: from='client.? 172.21.15.123:0/7128576' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-18T02:54:45.221 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-06-18T02:54:45.222 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph config generate-minimal-conf 2024-06-18T02:54:45.254 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:44 smithi016 ceph-mon[29214]: from='client.? 172.21.15.123:0/7128576' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-18T02:54:45.535 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T02:54:46.484 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:54:46 smithi123 ceph-mon[36358]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:54:46.485 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:54:46 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:54:46.485 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:54:46 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:54:46.485 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:54:46 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:54:46.485 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:54:46 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T02:54:46.485 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:54:46 smithi123 ceph-mon[36358]: Updating smithi016:/etc/ceph/ceph.conf 2024-06-18T02:54:46.485 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:54:46 smithi123 ceph-mon[36358]: Updating smithi123:/etc/ceph/ceph.conf 2024-06-18T02:54:46.485 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:54:46 smithi123 ceph-mon[36358]: Updating smithi123:/var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-18T02:54:46.485 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:54:46 smithi123 ceph-mon[36358]: Updating smithi016:/var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-18T02:54:46.485 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:54:46 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:54:46.485 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:54:46 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:54:46.485 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:54:46 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:54:46.485 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:54:46 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:54:46.485 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:54:46 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:54:46.486 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:54:46 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-18T02:54:46.486 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:54:46 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-18T02:54:46.486 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:54:46 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:54:46.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:46 smithi016 ceph-mon[29214]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:54:46.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:46 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:54:46.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:46 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:54:46.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:46 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:54:46.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:46 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T02:54:46.556 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:46 smithi016 ceph-mon[29214]: Updating smithi016:/etc/ceph/ceph.conf 2024-06-18T02:54:46.556 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:46 smithi016 ceph-mon[29214]: Updating smithi123:/etc/ceph/ceph.conf 2024-06-18T02:54:46.556 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:46 smithi016 ceph-mon[29214]: Updating smithi123:/var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-18T02:54:46.556 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:46 smithi016 ceph-mon[29214]: Updating smithi016:/var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-18T02:54:46.556 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:46 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:54:46.556 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:46 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:54:46.556 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:46 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:54:46.556 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:46 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:54:46.556 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:46 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:54:46.556 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:46 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-18T02:54:46.556 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:46 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-18T02:54:46.556 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:46 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:54:47.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:47 smithi016 ceph-mon[29214]: Reconfiguring mon.smithi016 (unknown last config time)... 2024-06-18T02:54:47.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:47 smithi016 ceph-mon[29214]: Reconfiguring daemon mon.smithi016 on smithi016 2024-06-18T02:54:47.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:47 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:54:47.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:54:47 smithi123 ceph-mon[36358]: Reconfiguring mon.smithi016 (unknown last config time)... 2024-06-18T02:54:47.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:54:47 smithi123 ceph-mon[36358]: Reconfiguring daemon mon.smithi016 on smithi016 2024-06-18T02:54:47.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:54:47 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:54:48.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:48.374+0000 7f32e05cb640 1 -- 172.21.15.16:0/1328961166 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32d81022f0 msgr2=0x7f32d81026c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:54:48.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:48.374+0000 7f32e05cb640 1 --2- 172.21.15.16:0/1328961166 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32d81022f0 0x7f32d81026c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f32d40048d0 tx=0x7f32d4009da0 comp rx=0 tx=0).stop 2024-06-18T02:54:48.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:48.374+0000 7f32e05cb640 1 -- 172.21.15.16:0/1328961166 shutdown_connections 2024-06-18T02:54:48.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:48.374+0000 7f32e05cb640 1 --2- 172.21.15.16:0/1328961166 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32d8102c90 0x7f32d80fa9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:54:48.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:48.374+0000 7f32e05cb640 1 --2- 172.21.15.16:0/1328961166 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32d81022f0 0x7f32d81026c0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:54:48.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:48.374+0000 7f32e05cb640 1 -- 172.21.15.16:0/1328961166 >> 172.21.15.16:0/1328961166 conn(0x7f32d8070a40 msgr2=0x7f32d8070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:54:48.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:48.374+0000 7f32e05cb640 1 -- 172.21.15.16:0/1328961166 shutdown_connections 2024-06-18T02:54:48.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:48.374+0000 7f32e05cb640 1 -- 172.21.15.16:0/1328961166 wait complete. 2024-06-18T02:54:48.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:48.374+0000 7f32e05cb640 1 Processor -- start 2024-06-18T02:54:48.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:48.375+0000 7f32e05cb640 1 -- start start 2024-06-18T02:54:48.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:48.375+0000 7f32e05cb640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32d81022f0 0x7f32d8115350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:54:48.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:48.375+0000 7f32e05cb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32d8102c90 0x7f32d8115890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:54:48.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:48.375+0000 7f32e05cb640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32d8115f20 con 0x7f32d8102c90 2024-06-18T02:54:48.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:48.375+0000 7f32e05cb640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32d810f2d0 con 0x7f32d81022f0 2024-06-18T02:54:48.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:48.375+0000 7f32ddb66640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32d8102c90 0x7f32d8115890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:54:48.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:48.375+0000 7f32ddb66640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32d8102c90 0x7f32d8115890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40156/0 (socket says 172.21.15.16:40156) 2024-06-18T02:54:48.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:48.375+0000 7f32ddb66640 1 -- 172.21.15.16:0/2667368892 learned_addr learned my addr 172.21.15.16:0/2667368892 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:54:48.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:48.375+0000 7f32de367640 1 --2- 172.21.15.16:0/2667368892 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32d81022f0 0x7f32d8115350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:54:48.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:48.375+0000 7f32ddb66640 1 -- 172.21.15.16:0/2667368892 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32d81022f0 msgr2=0x7f32d8115350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:54:48.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:48.375+0000 7f32ddb66640 1 --2- 172.21.15.16:0/2667368892 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32d81022f0 0x7f32d8115350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:54:48.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:48.375+0000 7f32ddb66640 1 -- 172.21.15.16:0/2667368892 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32d4005040 con 0x7f32d8102c90 2024-06-18T02:54:48.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:48.375+0000 7f32ddb66640 1 --2- 172.21.15.16:0/2667368892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32d8102c90 0x7f32d8115890 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f32c800b540 tx=0x7f32c80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:54:48.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:48.376+0000 7f32ceffd640 1 -- 172.21.15.16:0/2667368892 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f32c800f6f0 con 0x7f32d8102c90 2024-06-18T02:54:48.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:48.376+0000 7f32ceffd640 1 -- 172.21.15.16:0/2667368892 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f32c800fd20 con 0x7f32d8102c90 2024-06-18T02:54:48.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:48.376+0000 7f32e05cb640 1 -- 172.21.15.16:0/2667368892 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32d810f5b0 con 0x7f32d8102c90 2024-06-18T02:54:48.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:48.376+0000 7f32ceffd640 1 -- 172.21.15.16:0/2667368892 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f32c80103a0 con 0x7f32d8102c90 2024-06-18T02:54:48.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:48.376+0000 7f32e05cb640 1 -- 172.21.15.16:0/2667368892 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32d810fad0 con 0x7f32d8102c90 2024-06-18T02:54:48.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:48.377+0000 7f32e05cb640 1 -- 172.21.15.16:0/2667368892 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32d8065510 con 0x7f32d8102c90 2024-06-18T02:54:48.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:48.377+0000 7f32ceffd640 1 -- 172.21.15.16:0/2667368892 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) v1 ==== 83689+0+0 (secure 0 0 0) 0x7f32c8011400 con 0x7f32d8102c90 2024-06-18T02:54:48.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:48.377+0000 7f32ceffd640 1 --2- 172.21.15.16:0/2667368892 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f32c40653a0 0x7f32c4067850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:54:48.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:48.377+0000 7f32ceffd640 1 -- 172.21.15.16:0/2667368892 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f32c80816a0 con 0x7f32d8102c90 2024-06-18T02:54:48.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:48.377+0000 7f32de367640 1 --2- 172.21.15.16:0/2667368892 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f32c40653a0 0x7f32c4067850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:54:48.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:48.378+0000 7f32de367640 1 --2- 172.21.15.16:0/2667368892 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f32c40653a0 0x7f32c4067850 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f32d40043d0 tx=0x7f32d4009da0 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:54:48.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:48.382+0000 7f32ceffd640 1 -- 172.21.15.16:0/2667368892 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f32c8050970 con 0x7f32d8102c90 2024-06-18T02:54:48.545 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:48 smithi016 ceph-mon[29214]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:54:48.545 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:48 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:54:48.545 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:48 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:54:48.545 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:48 smithi016 ceph-mon[29214]: Reconfiguring mgr.smithi016.amxydz (unknown last config time)... 2024-06-18T02:54:48.546 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:48 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi016.amxydz", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-18T02:54:48.546 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:48 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-18T02:54:48.546 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:48 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:54:48.546 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:48 smithi016 ceph-mon[29214]: Reconfiguring daemon mgr.smithi016.amxydz on smithi016 2024-06-18T02:54:48.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:48.555+0000 7f32e05cb640 1 -- 172.21.15.16:0/2667368892 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f32d8074960 con 0x7f32d8102c90 2024-06-18T02:54:48.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:48.556+0000 7f32ceffd640 1 -- 172.21.15.16:0/2667368892 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v10) v1 ==== 76+0+225 (secure 0 0 0) 0x7f32c8080e50 con 0x7f32d8102c90 2024-06-18T02:54:48.556 INFO:teuthology.orchestra.run.smithi016.stdout:# minimal ceph.conf for 9e93709a-2d1d-11ef-bc9d-c7b262605968 2024-06-18T02:54:48.556 INFO:teuthology.orchestra.run.smithi016.stdout:[global] 2024-06-18T02:54:48.556 INFO:teuthology.orchestra.run.smithi016.stdout: fsid = 9e93709a-2d1d-11ef-bc9d-c7b262605968 2024-06-18T02:54:48.556 INFO:teuthology.orchestra.run.smithi016.stdout: mon_host = [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] 2024-06-18T02:54:48.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:48.558+0000 7f32e05cb640 1 -- 172.21.15.16:0/2667368892 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f32c40653a0 msgr2=0x7f32c4067850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:54:48.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:48.558+0000 7f32e05cb640 1 --2- 172.21.15.16:0/2667368892 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f32c40653a0 0x7f32c4067850 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f32d40043d0 tx=0x7f32d4009da0 comp rx=0 tx=0).stop 2024-06-18T02:54:48.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:48.558+0000 7f32e05cb640 1 -- 172.21.15.16:0/2667368892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32d8102c90 msgr2=0x7f32d8115890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:54:48.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:48.558+0000 7f32e05cb640 1 --2- 172.21.15.16:0/2667368892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32d8102c90 0x7f32d8115890 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f32c800b540 tx=0x7f32c80091a0 comp rx=0 tx=0).stop 2024-06-18T02:54:48.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:48.558+0000 7f32e05cb640 1 -- 172.21.15.16:0/2667368892 shutdown_connections 2024-06-18T02:54:48.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:48.558+0000 7f32e05cb640 1 --2- 172.21.15.16:0/2667368892 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f32c40653a0 0x7f32c4067850 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:54:48.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:48.558+0000 7f32e05cb640 1 --2- 172.21.15.16:0/2667368892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32d8102c90 0x7f32d8115890 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:54:48.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:48.558+0000 7f32e05cb640 1 --2- 172.21.15.16:0/2667368892 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32d81022f0 0x7f32d8115350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:54:48.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:48.558+0000 7f32e05cb640 1 -- 172.21.15.16:0/2667368892 >> 172.21.15.16:0/2667368892 conn(0x7f32d8070a40 msgr2=0x7f32d81030a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:54:48.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:48.558+0000 7f32e05cb640 1 -- 172.21.15.16:0/2667368892 shutdown_connections 2024-06-18T02:54:48.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:48.558+0000 7f32e05cb640 1 -- 172.21.15.16:0/2667368892 wait complete. 2024-06-18T02:54:48.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:54:48 smithi123 ceph-mon[36358]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:54:48.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:54:48 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:54:48.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:54:48 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:54:48.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:54:48 smithi123 ceph-mon[36358]: Reconfiguring mgr.smithi016.amxydz (unknown last config time)... 2024-06-18T02:54:48.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:54:48 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi016.amxydz", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-18T02:54:48.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:54:48 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-18T02:54:48.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:54:48 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:54:48.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:54:48 smithi123 ceph-mon[36358]: Reconfiguring daemon mgr.smithi016.amxydz on smithi016 2024-06-18T02:54:49.458 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:49 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/2667368892' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:54:49.487 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-06-18T02:54:49.487 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-06-18T02:54:49.487 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/ceph/ceph.conf 2024-06-18T02:54:49.558 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-06-18T02:54:49.558 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-06-18T02:54:49.625 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-06-18T02:54:49.626 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/etc/ceph/ceph.conf 2024-06-18T02:54:49.652 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:54:49 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/2667368892' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:54:49.658 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-06-18T02:54:49.658 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-06-18T02:54:49.728 INFO:tasks.cephadm:Deploying OSDs... 2024-06-18T02:54:49.729 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-06-18T02:54:49.729 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/scratch_devs of=/dev/stdout 2024-06-18T02:54:49.746 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-06-18T02:54:49.746 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_1 2024-06-18T02:54:49.803 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-06-18T02:54:49.803 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-18T02:54:49.803 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 676 Links: 1 2024-06-18T02:54:49.803 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-18T02:54:49.803 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2024-06-18T02:54:49.803 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-06-18 02:54:04.868344446 +0000 2024-06-18T02:54:49.803 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-06-18 02:48:10.059135042 +0000 2024-06-18T02:54:49.803 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-06-18 02:48:10.059135042 +0000 2024-06-18T02:54:49.803 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-06-18 02:48:10.059135042 +0000 2024-06-18T02:54:49.803 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-06-18T02:54:49.868 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-06-18T02:54:49.869 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-06-18T02:54:49.869 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000114063 s, 4.5 MB/s 2024-06-18T02:54:49.870 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-06-18T02:54:49.930 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_2 2024-06-18T02:54:49.986 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-06-18T02:54:49.986 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-18T02:54:49.986 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 693 Links: 1 2024-06-18T02:54:49.986 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-18T02:54:49.986 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2024-06-18T02:54:49.986 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-06-18 02:54:04.869344430 +0000 2024-06-18T02:54:49.986 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-06-18 02:48:10.390129728 +0000 2024-06-18T02:54:49.986 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-06-18 02:48:10.390129728 +0000 2024-06-18T02:54:49.987 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-06-18 02:48:10.390129728 +0000 2024-06-18T02:54:49.987 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-06-18T02:54:50.052 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-06-18T02:54:50.052 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-06-18T02:54:50.052 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000108323 s, 4.7 MB/s 2024-06-18T02:54:50.053 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-06-18T02:54:50.111 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_3 2024-06-18T02:54:50.167 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-06-18T02:54:50.168 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-18T02:54:50.168 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 712 Links: 1 2024-06-18T02:54:50.168 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-18T02:54:50.168 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2024-06-18T02:54:50.168 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-06-18 02:54:04.869344430 +0000 2024-06-18T02:54:50.169 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-06-18 02:48:10.736124173 +0000 2024-06-18T02:54:50.169 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-06-18 02:48:10.736124173 +0000 2024-06-18T02:54:50.169 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-06-18 02:48:10.736124173 +0000 2024-06-18T02:54:50.169 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-06-18T02:54:50.233 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-06-18T02:54:50.233 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-06-18T02:54:50.233 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 8.9403e-05 s, 5.7 MB/s 2024-06-18T02:54:50.234 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-06-18T02:54:50.293 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_4 2024-06-18T02:54:50.350 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-06-18T02:54:50.350 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-18T02:54:50.350 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 730 Links: 1 2024-06-18T02:54:50.350 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-18T02:54:50.350 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2024-06-18T02:54:50.350 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-06-18 02:54:04.870344414 +0000 2024-06-18T02:54:50.351 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-06-18 02:48:11.049119148 +0000 2024-06-18T02:54:50.351 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-06-18 02:48:11.049119148 +0000 2024-06-18T02:54:50.351 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-06-18 02:48:11.049119148 +0000 2024-06-18T02:54:50.351 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-06-18T02:54:50.414 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-06-18T02:54:50.414 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-06-18T02:54:50.414 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 8.8176e-05 s, 5.8 MB/s 2024-06-18T02:54:50.415 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-06-18T02:54:50.472 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-06-18T02:54:50.472 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/scratch_devs of=/dev/stdout 2024-06-18T02:54:50.492 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-06-18T02:54:50.492 DEBUG:teuthology.orchestra.run.smithi123:> stat /dev/vg_nvme/lv_1 2024-06-18T02:54:50.550 INFO:teuthology.orchestra.run.smithi123.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-06-18T02:54:50.551 INFO:teuthology.orchestra.run.smithi123.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-18T02:54:50.551 INFO:teuthology.orchestra.run.smithi123.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-06-18T02:54:50.551 INFO:teuthology.orchestra.run.smithi123.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-18T02:54:50.551 INFO:teuthology.orchestra.run.smithi123.stdout:Context: system_u:object_r:device_t:s0 2024-06-18T02:54:50.551 INFO:teuthology.orchestra.run.smithi123.stdout:Access: 2024-06-18 02:54:36.093861473 +0000 2024-06-18T02:54:50.551 INFO:teuthology.orchestra.run.smithi123.stdout:Modify: 2024-06-18 02:47:59.539119789 +0000 2024-06-18T02:54:50.551 INFO:teuthology.orchestra.run.smithi123.stdout:Change: 2024-06-18 02:47:59.539119789 +0000 2024-06-18T02:54:50.551 INFO:teuthology.orchestra.run.smithi123.stdout: Birth: 2024-06-18 02:47:59.539119789 +0000 2024-06-18T02:54:50.552 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-06-18T02:54:50.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:50 smithi016 ceph-mon[29214]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:54:50.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:50 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:54:50.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:50 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:54:50.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:50 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi016", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-06-18T02:54:50.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:50 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:54:50.614 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:54:50 smithi123 ceph-mon[36358]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:54:50.615 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:54:50 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:54:50.615 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:54:50 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:54:50.615 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:54:50 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi016", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-06-18T02:54:50.615 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:54:50 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:54:50.617 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2024-06-18T02:54:50.617 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2024-06-18T02:54:50.617 INFO:teuthology.orchestra.run.smithi123.stderr:512 bytes copied, 0.000236925 s, 2.2 MB/s 2024-06-18T02:54:50.619 DEBUG:teuthology.orchestra.run.smithi123:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-06-18T02:54:50.677 DEBUG:teuthology.orchestra.run.smithi123:> stat /dev/vg_nvme/lv_2 2024-06-18T02:54:50.736 INFO:teuthology.orchestra.run.smithi123.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-06-18T02:54:50.736 INFO:teuthology.orchestra.run.smithi123.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-18T02:54:50.736 INFO:teuthology.orchestra.run.smithi123.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-06-18T02:54:50.736 INFO:teuthology.orchestra.run.smithi123.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-18T02:54:50.736 INFO:teuthology.orchestra.run.smithi123.stdout:Context: system_u:object_r:device_t:s0 2024-06-18T02:54:50.736 INFO:teuthology.orchestra.run.smithi123.stdout:Access: 2024-06-18 02:54:36.094861448 +0000 2024-06-18T02:54:50.736 INFO:teuthology.orchestra.run.smithi123.stdout:Modify: 2024-06-18 02:47:59.861111769 +0000 2024-06-18T02:54:50.737 INFO:teuthology.orchestra.run.smithi123.stdout:Change: 2024-06-18 02:47:59.861111769 +0000 2024-06-18T02:54:50.737 INFO:teuthology.orchestra.run.smithi123.stdout: Birth: 2024-06-18 02:47:59.861111769 +0000 2024-06-18T02:54:50.737 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-06-18T02:54:50.802 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2024-06-18T02:54:50.802 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2024-06-18T02:54:50.802 INFO:teuthology.orchestra.run.smithi123.stderr:512 bytes copied, 0.000275328 s, 1.9 MB/s 2024-06-18T02:54:50.803 DEBUG:teuthology.orchestra.run.smithi123:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-06-18T02:54:50.861 DEBUG:teuthology.orchestra.run.smithi123:> stat /dev/vg_nvme/lv_3 2024-06-18T02:54:50.917 INFO:teuthology.orchestra.run.smithi123.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-06-18T02:54:50.917 INFO:teuthology.orchestra.run.smithi123.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-18T02:54:50.917 INFO:teuthology.orchestra.run.smithi123.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-06-18T02:54:50.917 INFO:teuthology.orchestra.run.smithi123.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-18T02:54:50.917 INFO:teuthology.orchestra.run.smithi123.stdout:Context: system_u:object_r:device_t:s0 2024-06-18T02:54:50.917 INFO:teuthology.orchestra.run.smithi123.stdout:Access: 2024-06-18 02:54:36.094861448 +0000 2024-06-18T02:54:50.917 INFO:teuthology.orchestra.run.smithi123.stdout:Modify: 2024-06-18 02:48:00.194103474 +0000 2024-06-18T02:54:50.917 INFO:teuthology.orchestra.run.smithi123.stdout:Change: 2024-06-18 02:48:00.194103474 +0000 2024-06-18T02:54:50.917 INFO:teuthology.orchestra.run.smithi123.stdout: Birth: 2024-06-18 02:48:00.194103474 +0000 2024-06-18T02:54:50.917 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-06-18T02:54:50.981 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2024-06-18T02:54:50.981 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2024-06-18T02:54:50.981 INFO:teuthology.orchestra.run.smithi123.stderr:512 bytes copied, 0.00019908 s, 2.6 MB/s 2024-06-18T02:54:50.982 DEBUG:teuthology.orchestra.run.smithi123:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-06-18T02:54:51.039 DEBUG:teuthology.orchestra.run.smithi123:> stat /dev/vg_nvme/lv_4 2024-06-18T02:54:51.095 INFO:teuthology.orchestra.run.smithi123.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-06-18T02:54:51.095 INFO:teuthology.orchestra.run.smithi123.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-18T02:54:51.095 INFO:teuthology.orchestra.run.smithi123.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-06-18T02:54:51.096 INFO:teuthology.orchestra.run.smithi123.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-18T02:54:51.096 INFO:teuthology.orchestra.run.smithi123.stdout:Context: system_u:object_r:device_t:s0 2024-06-18T02:54:51.096 INFO:teuthology.orchestra.run.smithi123.stdout:Access: 2024-06-18 02:54:36.094861448 +0000 2024-06-18T02:54:51.096 INFO:teuthology.orchestra.run.smithi123.stdout:Modify: 2024-06-18 02:48:00.511095578 +0000 2024-06-18T02:54:51.096 INFO:teuthology.orchestra.run.smithi123.stdout:Change: 2024-06-18 02:48:00.511095578 +0000 2024-06-18T02:54:51.096 INFO:teuthology.orchestra.run.smithi123.stdout: Birth: 2024-06-18 02:48:00.511095578 +0000 2024-06-18T02:54:51.096 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-06-18T02:54:51.158 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2024-06-18T02:54:51.159 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2024-06-18T02:54:51.159 INFO:teuthology.orchestra.run.smithi123.stderr:512 bytes copied, 0.0001774 s, 2.9 MB/s 2024-06-18T02:54:51.161 DEBUG:teuthology.orchestra.run.smithi123:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-06-18T02:54:51.224 INFO:tasks.cephadm:Deploying osd.0 on smithi016 with /dev/vg_nvme/lv_4... 2024-06-18T02:54:51.225 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-06-18T02:54:51.520 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T02:54:51.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:51 smithi016 ceph-mon[29214]: Reconfiguring crash.smithi016 (monmap changed)... 2024-06-18T02:54:51.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:51 smithi016 ceph-mon[29214]: Reconfiguring daemon crash.smithi016 on smithi016 2024-06-18T02:54:51.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:54:51 smithi123 ceph-mon[36358]: Reconfiguring crash.smithi016 (monmap changed)... 2024-06-18T02:54:51.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:54:51 smithi123 ceph-mon[36358]: Reconfiguring daemon crash.smithi016 on smithi016 2024-06-18T02:54:52.551 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:52 smithi016 ceph-mon[29214]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:54:52.551 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:52 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:54:52.551 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:52 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:54:52.551 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:52 smithi016 ceph-mon[29214]: Reconfiguring alertmanager.smithi016 (dependencies changed)... 2024-06-18T02:54:52.551 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:52 smithi016 ceph-mon[29214]: Reconfiguring daemon alertmanager.smithi016 on smithi016 2024-06-18T02:54:52.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:54:52 smithi123 ceph-mon[36358]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:54:52.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:54:52 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:54:52.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:54:52 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:54:52.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:54:52 smithi123 ceph-mon[36358]: Reconfiguring alertmanager.smithi016 (dependencies changed)... 2024-06-18T02:54:52.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:54:52 smithi123 ceph-mon[36358]: Reconfiguring daemon alertmanager.smithi016 on smithi016 2024-06-18T02:54:53.990 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-06-18T02:54:54.102 INFO:teuthology.orchestra.run.smithi016.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-06-18T02:54:54.113 INFO:teuthology.orchestra.run.smithi016.stderr: stderr: 10+0 records in 2024-06-18T02:54:54.113 INFO:teuthology.orchestra.run.smithi016.stderr:10+0 records out 2024-06-18T02:54:54.113 INFO:teuthology.orchestra.run.smithi016.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00847518 s, 1.2 GB/s 2024-06-18T02:54:54.113 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping successful for: 2024-06-18T02:54:54.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:54 smithi016 ceph-mon[29214]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:54:54.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:54:54 smithi123 ceph-mon[36358]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:54:55.232 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph orch daemon add osd smithi016:vg_nvme/lv_4 2024-06-18T02:54:56.375 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T02:54:56.491 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:56 smithi016 ceph-mon[29214]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:54:56.491 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:56 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:54:56.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:54:56 smithi123 ceph-mon[36358]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:54:56.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:54:56 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:54:57.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:57.883+0000 7fef254a8640 1 -- 172.21.15.16:0/2796068922 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef200fc2b0 msgr2=0x7fef200fc6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:54:57.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:57.883+0000 7fef254a8640 1 --2- 172.21.15.16:0/2796068922 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef200fc2b0 0x7fef200fc6c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fef100048d0 tx=0x7fef10009da0 comp rx=0 tx=0).stop 2024-06-18T02:54:57.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:57.883+0000 7fef254a8640 1 -- 172.21.15.16:0/2796068922 shutdown_connections 2024-06-18T02:54:57.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:57.883+0000 7fef254a8640 1 --2- 172.21.15.16:0/2796068922 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef200fd580 0x7fef200fd9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:54:57.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:57.883+0000 7fef254a8640 1 --2- 172.21.15.16:0/2796068922 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef200fc2b0 0x7fef200fc6c0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:54:57.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:57.883+0000 7fef254a8640 1 -- 172.21.15.16:0/2796068922 >> 172.21.15.16:0/2796068922 conn(0x7fef200f7a90 msgr2=0x7fef200f9ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:54:57.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:57.883+0000 7fef254a8640 1 -- 172.21.15.16:0/2796068922 shutdown_connections 2024-06-18T02:54:57.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:57.883+0000 7fef254a8640 1 -- 172.21.15.16:0/2796068922 wait complete. 2024-06-18T02:54:57.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:57.884+0000 7fef254a8640 1 Processor -- start 2024-06-18T02:54:57.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:57.884+0000 7fef254a8640 1 -- start start 2024-06-18T02:54:57.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:57.884+0000 7fef254a8640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef200fc2b0 0x7fef2010cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:54:57.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:57.884+0000 7fef254a8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef200fd580 0x7fef2010d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:54:57.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:57.884+0000 7fef254a8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef2010dad0 con 0x7fef200fd580 2024-06-18T02:54:57.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:57.884+0000 7fef254a8640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef2010e2a0 con 0x7fef200fc2b0 2024-06-18T02:54:57.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:57.885+0000 7fef1e7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef200fd580 0x7fef2010d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:54:57.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:57.885+0000 7fef1e7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef200fd580 0x7fef2010d480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35074/0 (socket says 172.21.15.16:35074) 2024-06-18T02:54:57.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:57.885+0000 7fef1e7fc640 1 -- 172.21.15.16:0/3050529017 learned_addr learned my addr 172.21.15.16:0/3050529017 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:54:57.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:57.885+0000 7fef1effd640 1 --2- 172.21.15.16:0/3050529017 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef200fc2b0 0x7fef2010cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:54:57.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:57.885+0000 7fef1e7fc640 1 -- 172.21.15.16:0/3050529017 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef200fc2b0 msgr2=0x7fef2010cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:54:57.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:57.885+0000 7fef1e7fc640 1 --2- 172.21.15.16:0/3050529017 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef200fc2b0 0x7fef2010cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:54:57.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:57.885+0000 7fef1e7fc640 1 -- 172.21.15.16:0/3050529017 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef10005040 con 0x7fef200fd580 2024-06-18T02:54:57.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:57.885+0000 7fef1e7fc640 1 --2- 172.21.15.16:0/3050529017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef200fd580 0x7fef2010d480 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fef14002a50 tx=0x7fef14004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:54:57.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:57.885+0000 7fef077fe640 1 -- 172.21.15.16:0/3050529017 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef14007c30 con 0x7fef200fd580 2024-06-18T02:54:57.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:57.885+0000 7fef077fe640 1 -- 172.21.15.16:0/3050529017 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fef1400f040 con 0x7fef200fd580 2024-06-18T02:54:57.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:57.885+0000 7fef077fe640 1 -- 172.21.15.16:0/3050529017 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef1400db50 con 0x7fef200fd580 2024-06-18T02:54:57.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:57.885+0000 7fef254a8640 1 -- 172.21.15.16:0/3050529017 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef2010e530 con 0x7fef200fd580 2024-06-18T02:54:57.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:57.885+0000 7fef254a8640 1 -- 172.21.15.16:0/3050529017 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef2010ead0 con 0x7fef200fd580 2024-06-18T02:54:57.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:57.886+0000 7fef254a8640 1 -- 172.21.15.16:0/3050529017 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef200fb710 con 0x7fef200fd580 2024-06-18T02:54:57.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:57.887+0000 7fef077fe640 1 -- 172.21.15.16:0/3050529017 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) v1 ==== 83689+0+0 (secure 0 0 0) 0x7fef1401c400 con 0x7fef200fd580 2024-06-18T02:54:57.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:57.887+0000 7fef077fe640 1 --2- 172.21.15.16:0/3050529017 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fef080653a0 0x7fef08067850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:54:57.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:57.887+0000 7fef077fe640 1 -- 172.21.15.16:0/3050529017 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fef14081020 con 0x7fef200fd580 2024-06-18T02:54:57.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:57.887+0000 7fef1effd640 1 --2- 172.21.15.16:0/3050529017 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fef080653a0 0x7fef08067850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:54:57.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:57.888+0000 7fef1effd640 1 --2- 172.21.15.16:0/3050529017 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fef080653a0 0x7fef08067850 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fef10000c00 tx=0x7fef10035040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:54:57.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:57.892+0000 7fef077fe640 1 -- 172.21.15.16:0/3050529017 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fef14050270 con 0x7fef200fd580 2024-06-18T02:54:58.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:54:58.075+0000 7fef254a8640 1 -- 172.21.15.16:0/3050529017 --> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fef2010ee00 con 0x7fef080653a0 2024-06-18T02:54:58.529 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:58 smithi016 ceph-mon[29214]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:54:58.530 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:58 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-18T02:54:58.530 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:58 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-18T02:54:58.530 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:58 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:54:58.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:54:58 smithi123 ceph-mon[36358]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:54:58.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:54:58 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-18T02:54:58.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:54:58 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-18T02:54:58.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:54:58 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:54:59.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:54:59 smithi016 ceph-mon[29214]: from='client.14252 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-18T02:54:59.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:54:59 smithi123 ceph-mon[36358]: from='client.14252 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-18T02:55:00.537 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:00 smithi016 ceph-mon[29214]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:55:00.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:00 smithi123 ceph-mon[36358]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:55:01.639 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:01 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:01.640 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:01 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:01.640 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:01 smithi016 ceph-mon[29214]: Reconfiguring grafana.smithi016 (dependencies changed)... 2024-06-18T02:55:01.640 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:01 smithi016 ceph-mon[29214]: Reconfiguring daemon grafana.smithi016 on smithi016 2024-06-18T02:55:01.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:01 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:01.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:01 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:01.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:01 smithi123 ceph-mon[36358]: Reconfiguring grafana.smithi016 (dependencies changed)... 2024-06-18T02:55:01.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:01 smithi123 ceph-mon[36358]: Reconfiguring daemon grafana.smithi016 on smithi016 2024-06-18T02:55:02.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:02 smithi016 ceph-mon[29214]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:55:02.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:02 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/3355777757' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75595aa3-a41d-4081-9e6a-e564de1b58ca"}]: dispatch 2024-06-18T02:55:02.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:02 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/3355777757' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "75595aa3-a41d-4081-9e6a-e564de1b58ca"}]': finished 2024-06-18T02:55:02.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:02 smithi016 ceph-mon[29214]: osdmap e6: 1 total, 0 up, 1 in 2024-06-18T02:55:02.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:02 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-18T02:55:02.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:02 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/947564955' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-18T02:55:02.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:02 smithi123 ceph-mon[36358]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:55:02.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:02 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/3355777757' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75595aa3-a41d-4081-9e6a-e564de1b58ca"}]: dispatch 2024-06-18T02:55:02.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:02 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/3355777757' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "75595aa3-a41d-4081-9e6a-e564de1b58ca"}]': finished 2024-06-18T02:55:02.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:02 smithi123 ceph-mon[36358]: osdmap e6: 1 total, 0 up, 1 in 2024-06-18T02:55:02.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:02 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-18T02:55:02.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:02 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/947564955' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-18T02:55:04.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:04 smithi123 ceph-mon[36358]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:55:04.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:04 smithi016 ceph-mon[29214]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:55:06.681 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:06 smithi016 ceph-mon[29214]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:55:06.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:06 smithi123 ceph-mon[36358]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:55:08.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:08 smithi123 ceph-mon[36358]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:55:08.757 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:08 smithi016 ceph-mon[29214]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:55:10.413 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:10 smithi016 ceph-mon[29214]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:55:10.413 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:10 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:10.413 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:10 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:10.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:10 smithi123 ceph-mon[36358]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:55:10.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:10 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:10.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:10 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:11.633 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:11 smithi016 ceph-mon[29214]: Reconfiguring prometheus.smithi016 (dependencies changed)... 2024-06-18T02:55:11.633 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:11 smithi016 ceph-mon[29214]: Reconfiguring daemon prometheus.smithi016 on smithi016 2024-06-18T02:55:11.633 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:11 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:55:11.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:11 smithi123 ceph-mon[36358]: Reconfiguring prometheus.smithi016 (dependencies changed)... 2024-06-18T02:55:11.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:11 smithi123 ceph-mon[36358]: Reconfiguring daemon prometheus.smithi016 on smithi016 2024-06-18T02:55:11.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:11 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:55:12.498 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:12 smithi016 ceph-mon[29214]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:55:12.498 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:12 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-06-18T02:55:12.498 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:12 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:55:12.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:12 smithi123 ceph-mon[36358]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:55:12.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:12 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-06-18T02:55:12.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:12 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:55:13.645 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:13 smithi016 ceph-mon[29214]: Deploying daemon osd.0 on smithi016 2024-06-18T02:55:13.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:13 smithi123 ceph-mon[36358]: Deploying daemon osd.0 on smithi016 2024-06-18T02:55:14.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:14 smithi123 ceph-mon[36358]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:55:14.745 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:14 smithi016 ceph-mon[29214]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:55:16.458 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:16 smithi123 ceph-mon[36358]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:55:16.458 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:16 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:16.458 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:16 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:16.458 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:16 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi123", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-06-18T02:55:16.458 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:16 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:55:16.664 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:16 smithi016 ceph-mon[29214]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:55:16.664 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:16 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:16.664 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:16 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:16.664 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:16 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi123", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-06-18T02:55:16.664 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:16 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:55:17.484 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:17 smithi123 ceph-mon[36358]: Reconfiguring crash.smithi123 (monmap changed)... 2024-06-18T02:55:17.485 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:17 smithi123 ceph-mon[36358]: Reconfiguring daemon crash.smithi123 on smithi123 2024-06-18T02:55:17.485 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:17 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:17.485 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:17 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:17.485 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:17 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi123.olpkyi", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-18T02:55:17.485 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:17 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-18T02:55:17.485 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:17 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:55:17.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:17 smithi016 ceph-mon[29214]: Reconfiguring crash.smithi123 (monmap changed)... 2024-06-18T02:55:17.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:17 smithi016 ceph-mon[29214]: Reconfiguring daemon crash.smithi123 on smithi123 2024-06-18T02:55:17.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:17 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:17.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:17 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:17.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:17 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi123.olpkyi", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-18T02:55:17.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:17 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-18T02:55:17.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:17 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:55:18.451 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:18 smithi016 ceph-mon[29214]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:55:18.451 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:18 smithi016 ceph-mon[29214]: Reconfiguring mgr.smithi123.olpkyi (monmap changed)... 2024-06-18T02:55:18.451 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:18 smithi016 ceph-mon[29214]: Reconfiguring daemon mgr.smithi123.olpkyi on smithi123 2024-06-18T02:55:18.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:18 smithi123 ceph-mon[36358]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:55:18.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:18 smithi123 ceph-mon[36358]: Reconfiguring mgr.smithi123.olpkyi (monmap changed)... 2024-06-18T02:55:18.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:18 smithi123 ceph-mon[36358]: Reconfiguring daemon mgr.smithi123.olpkyi on smithi123 2024-06-18T02:55:19.755 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:19 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:19.756 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:19 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:19.756 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:19 smithi016 ceph-mon[29214]: Reconfiguring mon.smithi123 (monmap changed)... 2024-06-18T02:55:19.756 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:19 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-18T02:55:19.756 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:19 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-18T02:55:19.756 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:19 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:55:19.756 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:19 smithi016 ceph-mon[29214]: Reconfiguring daemon mon.smithi123 on smithi123 2024-06-18T02:55:19.756 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:19 smithi016 ceph-mon[29214]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:55:19.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:19 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:19.985 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:19 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:19.985 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:19 smithi123 ceph-mon[36358]: Reconfiguring mon.smithi123 (monmap changed)... 2024-06-18T02:55:19.985 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:19 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-18T02:55:19.985 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:19 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-18T02:55:19.985 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:19 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:55:19.985 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:19 smithi123 ceph-mon[36358]: Reconfiguring daemon mon.smithi123 on smithi123 2024-06-18T02:55:19.985 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:19 smithi123 ceph-mon[36358]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:55:21.235 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:20 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:21.235 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:20 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:21.235 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:20 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-18T02:55:21.236 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:20 smithi123 ceph-mon[36358]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-18T02:55:21.236 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:20 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi016.front.sepia.ceph.com:9093"}]: dispatch 2024-06-18T02:55:21.236 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:20 smithi123 ceph-mon[36358]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi016.front.sepia.ceph.com:9093"}]: dispatch 2024-06-18T02:55:21.236 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:20 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:21.236 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:20 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-18T02:55:21.237 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:20 smithi123 ceph-mon[36358]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-18T02:55:21.237 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:20 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi016.front.sepia.ceph.com:3000"}]: dispatch 2024-06-18T02:55:21.237 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:20 smithi123 ceph-mon[36358]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi016.front.sepia.ceph.com:3000"}]: dispatch 2024-06-18T02:55:21.237 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:20 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:21.237 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:20 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-18T02:55:21.237 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:20 smithi123 ceph-mon[36358]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-18T02:55:21.237 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:20 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi016.front.sepia.ceph.com:9095"}]: dispatch 2024-06-18T02:55:21.237 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:20 smithi123 ceph-mon[36358]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi016.front.sepia.ceph.com:9095"}]: dispatch 2024-06-18T02:55:21.238 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:20 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:21.238 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:20 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:55:21.238 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:20 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:21.238 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:20 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:21.303 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:20 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:21.303 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:20 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:21.303 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:20 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-18T02:55:21.303 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:20 smithi016 ceph-mon[29214]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-18T02:55:21.303 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:20 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi016.front.sepia.ceph.com:9093"}]: dispatch 2024-06-18T02:55:21.304 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:20 smithi016 ceph-mon[29214]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi016.front.sepia.ceph.com:9093"}]: dispatch 2024-06-18T02:55:21.304 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:20 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:21.304 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:20 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-18T02:55:21.304 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:20 smithi016 ceph-mon[29214]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-18T02:55:21.304 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:20 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi016.front.sepia.ceph.com:3000"}]: dispatch 2024-06-18T02:55:21.304 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:20 smithi016 ceph-mon[29214]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi016.front.sepia.ceph.com:3000"}]: dispatch 2024-06-18T02:55:21.304 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:20 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:21.304 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:20 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-18T02:55:21.304 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:20 smithi016 ceph-mon[29214]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-18T02:55:21.304 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:20 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi016.front.sepia.ceph.com:9095"}]: dispatch 2024-06-18T02:55:21.304 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:20 smithi016 ceph-mon[29214]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi016.front.sepia.ceph.com:9095"}]: dispatch 2024-06-18T02:55:21.304 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:20 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:21.304 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:20 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:55:21.304 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:20 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:21.305 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:20 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:22.234 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:21 smithi123 ceph-mon[36358]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:55:22.234 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:21 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:22.234 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:21 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:22.305 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:21 smithi016 ceph-mon[29214]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:55:22.305 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:21 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:22.305 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:21 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:24.312 INFO:teuthology.orchestra.run.smithi016.stdout:Created osd(s) 0 on host 'smithi016' 2024-06-18T02:55:24.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:24.311+0000 7fef077fe640 1 -- 172.21.15.16:0/3050529017 <== mgr.14216 v2:172.21.15.16:6800/3814182425 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fef2010ee00 con 0x7fef080653a0 2024-06-18T02:55:24.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:24.314+0000 7fef254a8640 1 -- 172.21.15.16:0/3050529017 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fef080653a0 msgr2=0x7fef08067850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:55:24.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:24.314+0000 7fef254a8640 1 --2- 172.21.15.16:0/3050529017 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fef080653a0 0x7fef08067850 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fef10000c00 tx=0x7fef10035040 comp rx=0 tx=0).stop 2024-06-18T02:55:24.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:24.314+0000 7fef254a8640 1 -- 172.21.15.16:0/3050529017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef200fd580 msgr2=0x7fef2010d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:55:24.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:24.314+0000 7fef254a8640 1 --2- 172.21.15.16:0/3050529017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef200fd580 0x7fef2010d480 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fef14002a50 tx=0x7fef14004600 comp rx=0 tx=0).stop 2024-06-18T02:55:24.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:24.314+0000 7fef254a8640 1 -- 172.21.15.16:0/3050529017 shutdown_connections 2024-06-18T02:55:24.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:24.314+0000 7fef254a8640 1 --2- 172.21.15.16:0/3050529017 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fef080653a0 0x7fef08067850 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:55:24.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:24.314+0000 7fef254a8640 1 --2- 172.21.15.16:0/3050529017 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef200fd580 0x7fef2010d480 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:55:24.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:24.314+0000 7fef254a8640 1 --2- 172.21.15.16:0/3050529017 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef200fc2b0 0x7fef2010cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:55:24.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:24.314+0000 7fef254a8640 1 -- 172.21.15.16:0/3050529017 >> 172.21.15.16:0/3050529017 conn(0x7fef200f7a90 msgr2=0x7fef200f9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:55:24.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:24.314+0000 7fef254a8640 1 -- 172.21.15.16:0/3050529017 shutdown_connections 2024-06-18T02:55:24.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:24.314+0000 7fef254a8640 1 -- 172.21.15.16:0/3050529017 wait complete. 2024-06-18T02:55:24.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:24 smithi016 ceph-mon[29214]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:55:24.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:24 smithi123 ceph-mon[36358]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:55:24.862 DEBUG:teuthology.orchestra.run.smithi016:osd.0> sudo journalctl -f -n 0 -u ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968@osd.0.service 2024-06-18T02:55:24.864 INFO:tasks.cephadm:Deploying osd.1 on smithi016 with /dev/vg_nvme/lv_3... 2024-06-18T02:55:24.865 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-06-18T02:55:25.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:25 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:25.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:25 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:25.556 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:25 smithi016 ceph-mon[29214]: from='osd.0 [v2:172.21.15.16:6802/3634864038,v1:172.21.15.16:6803/3634864038]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-06-18T02:55:25.568 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T02:55:25.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:25 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:25.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:25 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:25.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:25 smithi123 ceph-mon[36358]: from='osd.0 [v2:172.21.15.16:6802/3634864038,v1:172.21.15.16:6803/3634864038]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-06-18T02:55:26.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:26 smithi016 ceph-mon[29214]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:55:26.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:26 smithi016 ceph-mon[29214]: from='osd.0 [v2:172.21.15.16:6802/3634864038,v1:172.21.15.16:6803/3634864038]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-06-18T02:55:26.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:26 smithi016 ceph-mon[29214]: osdmap e7: 1 total, 0 up, 1 in 2024-06-18T02:55:26.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:26 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-18T02:55:26.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:26 smithi016 ceph-mon[29214]: from='osd.0 [v2:172.21.15.16:6802/3634864038,v1:172.21.15.16:6803/3634864038]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-06-18T02:55:26.556 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:26 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:55:26.556 INFO:journalctl@ceph.osd.0.smithi016.stdout:Jun 18 02:55:26 smithi016 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-0[47382]: 2024-06-18T02:55:26.322+0000 7f6e3de1f640 -1 osd.0 0 waiting for initial osdmap 2024-06-18T02:55:26.556 INFO:journalctl@ceph.osd.0.smithi016.stdout:Jun 18 02:55:26 smithi016 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-0[47382]: 2024-06-18T02:55:26.326+0000 7f6e38c56640 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-18T02:55:26.701 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:26 smithi123 ceph-mon[36358]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:55:26.701 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:26 smithi123 ceph-mon[36358]: from='osd.0 [v2:172.21.15.16:6802/3634864038,v1:172.21.15.16:6803/3634864038]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-06-18T02:55:26.701 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:26 smithi123 ceph-mon[36358]: osdmap e7: 1 total, 0 up, 1 in 2024-06-18T02:55:26.701 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:26 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-18T02:55:26.701 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:26 smithi123 ceph-mon[36358]: from='osd.0 [v2:172.21.15.16:6802/3634864038,v1:172.21.15.16:6803/3634864038]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-06-18T02:55:26.702 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:26 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:55:27.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:27 smithi016 ceph-mon[29214]: from='osd.0 [v2:172.21.15.16:6802/3634864038,v1:172.21.15.16:6803/3634864038]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-06-18T02:55:27.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:27 smithi016 ceph-mon[29214]: osdmap e8: 1 total, 0 up, 1 in 2024-06-18T02:55:27.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:27 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-18T02:55:27.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:27 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-18T02:55:27.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:27 smithi123 ceph-mon[36358]: from='osd.0 [v2:172.21.15.16:6802/3634864038,v1:172.21.15.16:6803/3634864038]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-06-18T02:55:27.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:27 smithi123 ceph-mon[36358]: osdmap e8: 1 total, 0 up, 1 in 2024-06-18T02:55:27.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:27 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-18T02:55:27.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:27 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-18T02:55:27.763 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-06-18T02:55:27.851 INFO:teuthology.orchestra.run.smithi016.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-06-18T02:55:27.862 INFO:teuthology.orchestra.run.smithi016.stderr: stderr: 10+0 records in 2024-06-18T02:55:27.862 INFO:teuthology.orchestra.run.smithi016.stderr:10+0 records out 2024-06-18T02:55:27.862 INFO:teuthology.orchestra.run.smithi016.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0101593 s, 1.0 GB/s 2024-06-18T02:55:27.863 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping successful for: 2024-06-18T02:55:28.668 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:28 smithi016 ceph-mon[29214]: purged_snaps scrub starts 2024-06-18T02:55:28.668 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:28 smithi016 ceph-mon[29214]: purged_snaps scrub ok 2024-06-18T02:55:28.668 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:28 smithi016 ceph-mon[29214]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:55:28.668 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:28 smithi016 ceph-mon[29214]: osd.0 [v2:172.21.15.16:6802/3634864038,v1:172.21.15.16:6803/3634864038] boot 2024-06-18T02:55:28.669 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:28 smithi016 ceph-mon[29214]: osdmap e9: 1 total, 1 up, 1 in 2024-06-18T02:55:28.669 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:28 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-18T02:55:28.698 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph orch daemon add osd smithi016:vg_nvme/lv_3 2024-06-18T02:55:28.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:28 smithi123 ceph-mon[36358]: purged_snaps scrub starts 2024-06-18T02:55:28.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:28 smithi123 ceph-mon[36358]: purged_snaps scrub ok 2024-06-18T02:55:28.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:28 smithi123 ceph-mon[36358]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-18T02:55:28.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:28 smithi123 ceph-mon[36358]: osd.0 [v2:172.21.15.16:6802/3634864038,v1:172.21.15.16:6803/3634864038] boot 2024-06-18T02:55:28.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:28 smithi123 ceph-mon[36358]: osdmap e9: 1 total, 1 up, 1 in 2024-06-18T02:55:28.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:28 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-18T02:55:28.981 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T02:55:29.446 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:29 smithi016 ceph-mon[29214]: osdmap e10: 1 total, 1 up, 1 in 2024-06-18T02:55:29.447 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:29 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:29.447 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:29 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:29.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:29 smithi123 ceph-mon[36358]: osdmap e10: 1 total, 1 up, 1 in 2024-06-18T02:55:29.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:29 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:29.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:29 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:30.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:30.418+0000 7f14f8ffa640 1 -- 172.21.15.16:0/2624889742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14f40661d0 msgr2=0x7f14f40feef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:55:30.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:30.418+0000 7f14f8ffa640 1 --2- 172.21.15.16:0/2624889742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14f40661d0 0x7f14f40feef0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f14e40048d0 tx=0x7f14e4009da0 comp rx=0 tx=0).stop 2024-06-18T02:55:30.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:30.418+0000 7f14f8ffa640 1 -- 172.21.15.16:0/2624889742 shutdown_connections 2024-06-18T02:55:30.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:30.418+0000 7f14f8ffa640 1 --2- 172.21.15.16:0/2624889742 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14f40ff430 0x7f14f4101810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:55:30.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:30.418+0000 7f14f8ffa640 1 --2- 172.21.15.16:0/2624889742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14f40661d0 0x7f14f40feef0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:55:30.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:30.418+0000 7f14f8ffa640 1 -- 172.21.15.16:0/2624889742 >> 172.21.15.16:0/2624889742 conn(0x7f14f40f6960 msgr2=0x7f14f40f8d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:55:30.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:30.418+0000 7f14f8ffa640 1 -- 172.21.15.16:0/2624889742 shutdown_connections 2024-06-18T02:55:30.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:30.418+0000 7f14f8ffa640 1 -- 172.21.15.16:0/2624889742 wait complete. 2024-06-18T02:55:30.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:30.418+0000 7f14f8ffa640 1 Processor -- start 2024-06-18T02:55:30.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:30.419+0000 7f14f8ffa640 1 -- start start 2024-06-18T02:55:30.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:30.419+0000 7f14f8ffa640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14f40661d0 0x7f14f410cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:55:30.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:30.419+0000 7f14f8ffa640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14f40ff430 0x7f14f410d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:55:30.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:30.419+0000 7f14f8ffa640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14f410e310 con 0x7f14f40661d0 2024-06-18T02:55:30.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:30.419+0000 7f14f8ffa640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14f410e480 con 0x7f14f40ff430 2024-06-18T02:55:30.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:30.419+0000 7f14f259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14f40661d0 0x7f14f410cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:55:30.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:30.419+0000 7f14f259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14f40661d0 0x7f14f410cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47834/0 (socket says 172.21.15.16:47834) 2024-06-18T02:55:30.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:30.419+0000 7f14f259c640 1 -- 172.21.15.16:0/1928741599 learned_addr learned my addr 172.21.15.16:0/1928741599 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:55:30.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:30.419+0000 7f14f259c640 1 -- 172.21.15.16:0/1928741599 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14f40ff430 msgr2=0x7f14f410d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:55:30.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:30.419+0000 7f14f1d9b640 1 --2- 172.21.15.16:0/1928741599 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14f40ff430 0x7f14f410d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:55:30.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:30.419+0000 7f14f259c640 1 --2- 172.21.15.16:0/1928741599 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14f40ff430 0x7f14f410d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:55:30.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:30.419+0000 7f14f259c640 1 -- 172.21.15.16:0/1928741599 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14e4005040 con 0x7f14f40661d0 2024-06-18T02:55:30.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:30.419+0000 7f14f1d9b640 1 --2- 172.21.15.16:0/1928741599 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14f40ff430 0x7f14f410d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-18T02:55:30.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:30.419+0000 7f14f259c640 1 --2- 172.21.15.16:0/1928741599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14f40661d0 0x7f14f410cf60 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f14e4003080 tx=0x7f14e4009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:55:30.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:30.420+0000 7f14e2ffd640 1 -- 172.21.15.16:0/1928741599 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f14e4038070 con 0x7f14f40661d0 2024-06-18T02:55:30.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:30.420+0000 7f14f8ffa640 1 -- 172.21.15.16:0/1928741599 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14f410e700 con 0x7f14f40661d0 2024-06-18T02:55:30.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:30.420+0000 7f14e2ffd640 1 -- 172.21.15.16:0/1928741599 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f14e4032d60 con 0x7f14f40661d0 2024-06-18T02:55:30.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:30.420+0000 7f14e2ffd640 1 -- 172.21.15.16:0/1928741599 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f14e403dcd0 con 0x7f14f40661d0 2024-06-18T02:55:30.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:30.420+0000 7f14f8ffa640 1 -- 172.21.15.16:0/1928741599 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14f410ebc0 con 0x7f14f40661d0 2024-06-18T02:55:30.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:30.421+0000 7f14f8ffa640 1 -- 172.21.15.16:0/1928741599 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14f4105670 con 0x7f14f40661d0 2024-06-18T02:55:30.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:30.421+0000 7f14e2ffd640 1 -- 172.21.15.16:0/1928741599 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) v1 ==== 83689+0+0 (secure 0 0 0) 0x7f14e4030020 con 0x7f14f40661d0 2024-06-18T02:55:30.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:30.421+0000 7f14e2ffd640 1 --2- 172.21.15.16:0/1928741599 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f14dc0653a0 0x7f14dc067850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:55:30.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:30.422+0000 7f14e2ffd640 1 -- 172.21.15.16:0/1928741599 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1978+0+0 (secure 0 0 0) 0x7f14e40a2720 con 0x7f14f40661d0 2024-06-18T02:55:30.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:30.422+0000 7f14f1d9b640 1 --2- 172.21.15.16:0/1928741599 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f14dc0653a0 0x7f14dc067850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:55:30.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:30.422+0000 7f14f1d9b640 1 --2- 172.21.15.16:0/1928741599 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f14dc0653a0 0x7f14dc067850 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f14e8006fd0 tx=0x7f14e8008040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:55:30.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:30.426+0000 7f14e2ffd640 1 -- 172.21.15.16:0/1928741599 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f14e4071710 con 0x7f14f40661d0 2024-06-18T02:55:30.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:30 smithi016 ceph-mon[29214]: pgmap v37: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-18T02:55:30.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:30.612+0000 7f14f8ffa640 1 -- 172.21.15.16:0/1928741599 --> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f14f410ede0 con 0x7f14dc0653a0 2024-06-18T02:55:30.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:30 smithi123 ceph-mon[36358]: pgmap v37: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-18T02:55:31.644 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:31 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-18T02:55:31.644 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:31 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-18T02:55:31.644 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:31 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:55:31.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:31 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-18T02:55:31.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:31 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-18T02:55:31.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:31 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:55:32.518 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:32 smithi016 ceph-mon[29214]: from='client.14270 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-18T02:55:32.518 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:32 smithi016 ceph-mon[29214]: pgmap v38: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-18T02:55:32.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:32 smithi123 ceph-mon[36358]: from='client.14270 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-18T02:55:32.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:32 smithi123 ceph-mon[36358]: pgmap v38: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-18T02:55:34.515 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:34 smithi016 ceph-mon[29214]: pgmap v39: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-18T02:55:34.516 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:34 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:34.516 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:34 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:34.516 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:34 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-18T02:55:34.516 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:34 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:34.516 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:34 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:55:34.516 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:34 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T02:55:34.516 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:34 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:34.516 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:34 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:55:34.625 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:34 smithi123 ceph-mon[36358]: pgmap v39: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-18T02:55:34.625 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:34 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:34.625 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:34 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:34.626 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:34 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-18T02:55:34.626 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:34 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:34.626 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:34 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:55:34.626 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:34 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T02:55:34.626 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:34 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:34.626 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:34 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:55:35.504 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:35 smithi016 ceph-mon[29214]: Detected new or changed devices on smithi016 2024-06-18T02:55:35.504 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:35 smithi016 ceph-mon[29214]: Adjusting osd_memory_target on smithi016 to 12868M 2024-06-18T02:55:35.504 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:35 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/3805739438' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "33accf47-d8b6-4b00-b579-2897f47e381d"}]: dispatch 2024-06-18T02:55:35.505 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:35 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/3805739438' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "33accf47-d8b6-4b00-b579-2897f47e381d"}]': finished 2024-06-18T02:55:35.505 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:35 smithi016 ceph-mon[29214]: osdmap e11: 2 total, 1 up, 2 in 2024-06-18T02:55:35.505 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:35 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-18T02:55:35.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:35 smithi123 ceph-mon[36358]: Detected new or changed devices on smithi016 2024-06-18T02:55:35.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:35 smithi123 ceph-mon[36358]: Adjusting osd_memory_target on smithi016 to 12868M 2024-06-18T02:55:35.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:35 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/3805739438' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "33accf47-d8b6-4b00-b579-2897f47e381d"}]: dispatch 2024-06-18T02:55:35.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:35 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/3805739438' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "33accf47-d8b6-4b00-b579-2897f47e381d"}]': finished 2024-06-18T02:55:35.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:35 smithi123 ceph-mon[36358]: osdmap e11: 2 total, 1 up, 2 in 2024-06-18T02:55:35.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:35 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-18T02:55:36.492 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:36 smithi016 ceph-mon[29214]: pgmap v41: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-18T02:55:36.492 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:36 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/3899204616' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-18T02:55:36.492 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:36 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:36.492 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:36 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:36.700 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:36 smithi123 ceph-mon[36358]: pgmap v41: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-18T02:55:36.700 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:36 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/3899204616' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-18T02:55:36.700 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:36 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:36.700 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:36 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:38.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:38 smithi123 ceph-mon[36358]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-18T02:55:38.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:38 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:38.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:38 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:38.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:38 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:55:38.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:38 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T02:55:38.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:38 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:38.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:38 smithi016 ceph-mon[29214]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-18T02:55:38.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:38 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:38.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:38 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:38.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:38 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:55:38.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:38 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T02:55:38.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:38 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:40.464 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:40 smithi016 ceph-mon[29214]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-18T02:55:40.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:40 smithi123 ceph-mon[36358]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-18T02:55:41.455 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:41 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:55:41.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:41 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:55:42.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:42 smithi123 ceph-mon[36358]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-18T02:55:42.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:42 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-06-18T02:55:42.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:42 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:55:42.746 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:42 smithi016 ceph-mon[29214]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-18T02:55:42.746 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:42 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-06-18T02:55:42.746 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:42 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:55:43.643 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:43 smithi016 ceph-mon[29214]: Deploying daemon osd.1 on smithi016 2024-06-18T02:55:43.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:43 smithi123 ceph-mon[36358]: Deploying daemon osd.1 on smithi016 2024-06-18T02:55:44.629 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:44 smithi016 ceph-mon[29214]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-18T02:55:44.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:44 smithi123 ceph-mon[36358]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-18T02:55:46.700 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:46 smithi123 ceph-mon[36358]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-18T02:55:46.702 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:46 smithi016 ceph-mon[29214]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-18T02:55:47.603 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:47 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:55:47.603 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:47 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:47.603 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:47 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:47.688 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:47 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:55:47.689 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:47 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:47.689 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:47 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:48.484 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:48 smithi123 ceph-mon[36358]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-18T02:55:48.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:48 smithi016 ceph-mon[29214]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-18T02:55:49.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:49 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:49.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:49 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:49.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:49 smithi123 ceph-mon[36358]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-18T02:55:50.027 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:49 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:50.027 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:49 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:50.028 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:49 smithi016 ceph-mon[29214]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-18T02:55:50.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:50.069+0000 7f14e2ffd640 1 -- 172.21.15.16:0/1928741599 <== mgr.14216 v2:172.21.15.16:6800/3814182425 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f14f410ede0 con 0x7f14dc0653a0 2024-06-18T02:55:50.071 INFO:teuthology.orchestra.run.smithi016.stdout:Created osd(s) 1 on host 'smithi016' 2024-06-18T02:55:50.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:50.072+0000 7f14f8ffa640 1 -- 172.21.15.16:0/1928741599 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f14dc0653a0 msgr2=0x7f14dc067850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:55:50.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:50.072+0000 7f14f8ffa640 1 --2- 172.21.15.16:0/1928741599 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f14dc0653a0 0x7f14dc067850 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f14e8006fd0 tx=0x7f14e8008040 comp rx=0 tx=0).stop 2024-06-18T02:55:50.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:50.072+0000 7f14f8ffa640 1 -- 172.21.15.16:0/1928741599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14f40661d0 msgr2=0x7f14f410cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:55:50.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:50.072+0000 7f14f8ffa640 1 --2- 172.21.15.16:0/1928741599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14f40661d0 0x7f14f410cf60 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f14e4003080 tx=0x7f14e4009da0 comp rx=0 tx=0).stop 2024-06-18T02:55:50.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:50.072+0000 7f14f8ffa640 1 -- 172.21.15.16:0/1928741599 shutdown_connections 2024-06-18T02:55:50.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:50.072+0000 7f14f8ffa640 1 --2- 172.21.15.16:0/1928741599 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f14dc0653a0 0x7f14dc067850 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:55:50.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:50.072+0000 7f14f8ffa640 1 --2- 172.21.15.16:0/1928741599 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14f40ff430 0x7f14f410d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:55:50.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:50.072+0000 7f14f8ffa640 1 --2- 172.21.15.16:0/1928741599 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14f40661d0 0x7f14f410cf60 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:55:50.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:50.072+0000 7f14f8ffa640 1 -- 172.21.15.16:0/1928741599 >> 172.21.15.16:0/1928741599 conn(0x7f14f40f6960 msgr2=0x7f14f40f6d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:55:50.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:50.072+0000 7f14f8ffa640 1 -- 172.21.15.16:0/1928741599 shutdown_connections 2024-06-18T02:55:50.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:50.072+0000 7f14f8ffa640 1 -- 172.21.15.16:0/1928741599 wait complete. 2024-06-18T02:55:50.850 DEBUG:teuthology.orchestra.run.smithi016:osd.1> sudo journalctl -f -n 0 -u ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968@osd.1.service 2024-06-18T02:55:50.853 INFO:tasks.cephadm:Deploying osd.2 on smithi016 with /dev/vg_nvme/lv_2... 2024-06-18T02:55:50.853 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-06-18T02:55:51.176 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:51 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:51.176 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:51 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:51.177 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:51 smithi016 ceph-mon[29214]: from='osd.1 [v2:172.21.15.16:6810/500143512,v1:172.21.15.16:6811/500143512]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-06-18T02:55:51.484 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:51 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:51.484 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:51 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:51.484 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:51 smithi123 ceph-mon[36358]: from='osd.1 [v2:172.21.15.16:6810/500143512,v1:172.21.15.16:6811/500143512]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-06-18T02:55:52.305 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:52 smithi016 ceph-mon[29214]: from='osd.1 [v2:172.21.15.16:6810/500143512,v1:172.21.15.16:6811/500143512]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-06-18T02:55:52.305 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:52 smithi016 ceph-mon[29214]: osdmap e12: 2 total, 1 up, 2 in 2024-06-18T02:55:52.305 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:52 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-18T02:55:52.306 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:52 smithi016 ceph-mon[29214]: from='osd.1 [v2:172.21.15.16:6810/500143512,v1:172.21.15.16:6811/500143512]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-06-18T02:55:52.306 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:52 smithi016 ceph-mon[29214]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-18T02:55:52.306 INFO:journalctl@ceph.osd.1.smithi016.stdout:Jun 18 02:55:52 smithi016 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-1[54640]: 2024-06-18T02:55:52.075+0000 7f3a6b696640 -1 osd.1 0 waiting for initial osdmap 2024-06-18T02:55:52.306 INFO:journalctl@ceph.osd.1.smithi016.stdout:Jun 18 02:55:52 smithi016 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-1[54640]: 2024-06-18T02:55:52.080+0000 7f3a65ccc640 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-18T02:55:52.343 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T02:55:52.484 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:52 smithi123 ceph-mon[36358]: from='osd.1 [v2:172.21.15.16:6810/500143512,v1:172.21.15.16:6811/500143512]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-06-18T02:55:52.484 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:52 smithi123 ceph-mon[36358]: osdmap e12: 2 total, 1 up, 2 in 2024-06-18T02:55:52.484 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:52 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-18T02:55:52.485 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:52 smithi123 ceph-mon[36358]: from='osd.1 [v2:172.21.15.16:6810/500143512,v1:172.21.15.16:6811/500143512]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-06-18T02:55:52.485 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:52 smithi123 ceph-mon[36358]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-18T02:55:53.370 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:53 smithi016 ceph-mon[29214]: from='osd.1 [v2:172.21.15.16:6810/500143512,v1:172.21.15.16:6811/500143512]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-06-18T02:55:53.370 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:53 smithi016 ceph-mon[29214]: osdmap e13: 2 total, 1 up, 2 in 2024-06-18T02:55:53.370 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:53 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-18T02:55:53.370 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:53 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-18T02:55:53.370 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:53 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-18T02:55:53.484 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:53 smithi123 ceph-mon[36358]: from='osd.1 [v2:172.21.15.16:6810/500143512,v1:172.21.15.16:6811/500143512]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-06-18T02:55:53.484 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:53 smithi123 ceph-mon[36358]: osdmap e13: 2 total, 1 up, 2 in 2024-06-18T02:55:53.484 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:53 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-18T02:55:53.485 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:53 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-18T02:55:53.485 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:53 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-18T02:55:54.336 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:54 smithi016 ceph-mon[29214]: purged_snaps scrub starts 2024-06-18T02:55:54.336 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:54 smithi016 ceph-mon[29214]: purged_snaps scrub ok 2024-06-18T02:55:54.336 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:54 smithi016 ceph-mon[29214]: osd.1 [v2:172.21.15.16:6810/500143512,v1:172.21.15.16:6811/500143512] boot 2024-06-18T02:55:54.336 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:54 smithi016 ceph-mon[29214]: osdmap e14: 2 total, 2 up, 2 in 2024-06-18T02:55:54.336 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:54 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-18T02:55:54.336 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:54 smithi016 ceph-mon[29214]: pgmap v53: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-18T02:55:54.484 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:54 smithi123 ceph-mon[36358]: purged_snaps scrub starts 2024-06-18T02:55:54.484 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:54 smithi123 ceph-mon[36358]: purged_snaps scrub ok 2024-06-18T02:55:54.485 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:54 smithi123 ceph-mon[36358]: osd.1 [v2:172.21.15.16:6810/500143512,v1:172.21.15.16:6811/500143512] boot 2024-06-18T02:55:54.485 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:54 smithi123 ceph-mon[36358]: osdmap e14: 2 total, 2 up, 2 in 2024-06-18T02:55:54.485 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:54 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-18T02:55:54.485 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:54 smithi123 ceph-mon[36358]: pgmap v53: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-18T02:55:54.650 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-06-18T02:55:54.749 INFO:teuthology.orchestra.run.smithi016.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-06-18T02:55:54.759 INFO:teuthology.orchestra.run.smithi016.stderr: stderr: 10+0 records in 2024-06-18T02:55:54.759 INFO:teuthology.orchestra.run.smithi016.stderr:10+0 records out 2024-06-18T02:55:54.759 INFO:teuthology.orchestra.run.smithi016.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00837745 s, 1.3 GB/s 2024-06-18T02:55:54.759 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping successful for: 2024-06-18T02:55:55.305 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:55 smithi016 ceph-mon[29214]: osdmap e15: 2 total, 2 up, 2 in 2024-06-18T02:55:55.484 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:55 smithi123 ceph-mon[36358]: osdmap e15: 2 total, 2 up, 2 in 2024-06-18T02:55:55.608 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph orch daemon add osd smithi016:vg_nvme/lv_2 2024-06-18T02:55:55.881 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T02:55:56.300 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:56 smithi016 ceph-mon[29214]: pgmap v55: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-18T02:55:56.300 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:56 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:56.300 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:56 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:56.484 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:56 smithi123 ceph-mon[36358]: pgmap v55: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-18T02:55:56.485 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:56 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:56.485 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:56 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:55:57.305 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:57 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:55:57.484 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:57 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:55:57.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:57.754+0000 7f462ade8640 1 -- 172.21.15.16:0/2681862338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4624066470 msgr2=0x7f46240feee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:55:57.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:57.754+0000 7f462ade8640 1 --2- 172.21.15.16:0/2681862338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4624066470 0x7f46240feee0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f46180048d0 tx=0x7f4618009da0 comp rx=0 tx=0).stop 2024-06-18T02:55:57.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:57.754+0000 7f462ade8640 1 -- 172.21.15.16:0/2681862338 shutdown_connections 2024-06-18T02:55:57.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:57.754+0000 7f462ade8640 1 --2- 172.21.15.16:0/2681862338 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46240ff420 0x7f4624101800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:55:57.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:57.754+0000 7f462ade8640 1 --2- 172.21.15.16:0/2681862338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4624066470 0x7f46240feee0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:55:57.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:57.754+0000 7f462ade8640 1 -- 172.21.15.16:0/2681862338 >> 172.21.15.16:0/2681862338 conn(0x7f46240f6970 msgr2=0x7f46240f8dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:55:57.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:57.754+0000 7f462ade8640 1 -- 172.21.15.16:0/2681862338 shutdown_connections 2024-06-18T02:55:57.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:57.754+0000 7f462ade8640 1 -- 172.21.15.16:0/2681862338 wait complete. 2024-06-18T02:55:57.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:57.755+0000 7f462ade8640 1 Processor -- start 2024-06-18T02:55:57.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:57.755+0000 7f462ade8640 1 -- start start 2024-06-18T02:55:57.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:57.755+0000 7f462ade8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4624066470 0x7f46240fc8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:55:57.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:57.755+0000 7f462ade8640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46240ff420 0x7f46240fade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:55:57.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:57.755+0000 7f462ade8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46240fb370 con 0x7f4624066470 2024-06-18T02:55:57.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:57.755+0000 7f462ade8640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46240fb4b0 con 0x7f46240ff420 2024-06-18T02:55:57.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:57.755+0000 7f4628b84640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4624066470 0x7f46240fc8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:55:57.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:57.755+0000 7f4628b84640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4624066470 0x7f46240fc8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45994/0 (socket says 172.21.15.16:45994) 2024-06-18T02:55:57.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:57.755+0000 7f4628b84640 1 -- 172.21.15.16:0/3646379671 learned_addr learned my addr 172.21.15.16:0/3646379671 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:55:57.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:57.756+0000 7f4623fff640 1 --2- 172.21.15.16:0/3646379671 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46240ff420 0x7f46240fade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:55:57.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:57.756+0000 7f4628b84640 1 -- 172.21.15.16:0/3646379671 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46240ff420 msgr2=0x7f46240fade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:55:57.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:57.756+0000 7f4628b84640 1 --2- 172.21.15.16:0/3646379671 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46240ff420 0x7f46240fade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:55:57.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:57.756+0000 7f4628b84640 1 -- 172.21.15.16:0/3646379671 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4618005040 con 0x7f4624066470 2024-06-18T02:55:57.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:57.756+0000 7f4623fff640 1 --2- 172.21.15.16:0/3646379671 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46240ff420 0x7f46240fade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-18T02:55:57.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:57.756+0000 7f4628b84640 1 --2- 172.21.15.16:0/3646379671 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4624066470 0x7f46240fc8f0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f4618000c00 tx=0x7f4618009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:55:57.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:57.756+0000 7f46217fa640 1 -- 172.21.15.16:0/3646379671 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4618038070 con 0x7f4624066470 2024-06-18T02:55:57.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:57.756+0000 7f462ade8640 1 -- 172.21.15.16:0/3646379671 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46240fb730 con 0x7f4624066470 2024-06-18T02:55:57.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:57.756+0000 7f46217fa640 1 -- 172.21.15.16:0/3646379671 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4618032970 con 0x7f4624066470 2024-06-18T02:55:57.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:57.756+0000 7f46217fa640 1 -- 172.21.15.16:0/3646379671 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4618033950 con 0x7f4624066470 2024-06-18T02:55:57.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:57.756+0000 7f462ade8640 1 -- 172.21.15.16:0/3646379671 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4624114f00 con 0x7f4624066470 2024-06-18T02:55:57.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:57.757+0000 7f462ade8640 1 -- 172.21.15.16:0/3646379671 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46240fa630 con 0x7f4624066470 2024-06-18T02:55:57.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:57.757+0000 7f46217fa640 1 -- 172.21.15.16:0/3646379671 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 18) v1 ==== 83689+0+0 (secure 0 0 0) 0x7f4618030020 con 0x7f4624066470 2024-06-18T02:55:57.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:57.758+0000 7f46217fa640 1 --2- 172.21.15.16:0/3646379671 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f46140653a0 0x7f4614067850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:55:57.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:57.758+0000 7f46217fa640 1 -- 172.21.15.16:0/3646379671 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 2410+0+0 (secure 0 0 0) 0x7f46180a1b80 con 0x7f4624066470 2024-06-18T02:55:57.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:57.758+0000 7f4623fff640 1 --2- 172.21.15.16:0/3646379671 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f46140653a0 0x7f4614067850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:55:57.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:57.758+0000 7f4623fff640 1 --2- 172.21.15.16:0/3646379671 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f46140653a0 0x7f4614067850 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4610004c60 tx=0x7f4610009040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:55:57.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:57.762+0000 7f46217fa640 1 -- 172.21.15.16:0/3646379671 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f46180709c0 con 0x7f4624066470 2024-06-18T02:55:57.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:55:57.945+0000 7f462ade8640 1 -- 172.21.15.16:0/3646379671 --> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f46240f84d0 con 0x7f46140653a0 2024-06-18T02:55:58.305 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:58 smithi016 ceph-mon[29214]: pgmap v56: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-18T02:55:58.305 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:58 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-18T02:55:58.305 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:58 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-18T02:55:58.305 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:58 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:55:58.484 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:58 smithi123 ceph-mon[36358]: pgmap v56: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-18T02:55:58.484 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:58 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-18T02:55:58.484 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:58 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-18T02:55:58.485 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:58 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:55:59.360 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:55:59 smithi016 ceph-mon[29214]: from='client.14288 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-18T02:55:59.484 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:55:59 smithi123 ceph-mon[36358]: from='client.14288 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-18T02:56:00.290 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:00 smithi016 ceph-mon[29214]: pgmap v57: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-18T02:56:00.484 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:00 smithi123 ceph-mon[36358]: pgmap v57: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-18T02:56:02.038 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:01 smithi016 ceph-mon[29214]: Detected new or changed devices on smithi016 2024-06-18T02:56:02.038 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:01 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:56:02.039 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:01 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:56:02.039 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:01 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-18T02:56:02.039 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:01 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-18T02:56:02.039 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:01 smithi016 ceph-mon[29214]: Adjusting osd_memory_target on smithi016 to 6434M 2024-06-18T02:56:02.039 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:01 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:56:02.039 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:01 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:56:02.039 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:01 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T02:56:02.039 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:01 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:56:02.039 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:01 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:56:02.039 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:01 smithi016 ceph-mon[29214]: pgmap v58: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-18T02:56:02.234 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:01 smithi123 ceph-mon[36358]: Detected new or changed devices on smithi016 2024-06-18T02:56:02.235 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:01 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:56:02.235 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:01 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:56:02.235 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:01 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-18T02:56:02.235 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:01 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-18T02:56:02.235 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:01 smithi123 ceph-mon[36358]: Adjusting osd_memory_target on smithi016 to 6434M 2024-06-18T02:56:02.235 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:01 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:56:02.235 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:01 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:56:02.235 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:01 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T02:56:02.235 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:01 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:56:02.235 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:01 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:56:02.235 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:01 smithi123 ceph-mon[36358]: pgmap v58: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-18T02:56:03.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:02 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/3754562035' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f9d60224-3bc7-458c-a8b7-8384a7e219d6"}]: dispatch 2024-06-18T02:56:03.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:02 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/3754562035' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f9d60224-3bc7-458c-a8b7-8384a7e219d6"}]': finished 2024-06-18T02:56:03.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:02 smithi016 ceph-mon[29214]: osdmap e16: 3 total, 2 up, 3 in 2024-06-18T02:56:03.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:02 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-18T02:56:03.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:02 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/120278977' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-18T02:56:03.234 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:02 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/3754562035' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f9d60224-3bc7-458c-a8b7-8384a7e219d6"}]: dispatch 2024-06-18T02:56:03.234 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:02 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/3754562035' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f9d60224-3bc7-458c-a8b7-8384a7e219d6"}]': finished 2024-06-18T02:56:03.235 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:02 smithi123 ceph-mon[36358]: osdmap e16: 3 total, 2 up, 3 in 2024-06-18T02:56:03.235 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:02 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-18T02:56:03.235 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:02 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/120278977' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-18T02:56:04.234 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:03 smithi123 ceph-mon[36358]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-18T02:56:04.273 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:03 smithi016 ceph-mon[29214]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-18T02:56:06.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:06 smithi016 ceph-mon[29214]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-18T02:56:06.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:06 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:56:06.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:06 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:56:06.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:06 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:56:06.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:06 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T02:56:06.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:06 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:56:06.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:06 smithi123 ceph-mon[36358]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-18T02:56:06.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:06 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:56:06.985 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:06 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:56:06.985 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:06 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:56:06.985 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:06 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T02:56:06.985 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:06 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:56:08.657 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:08 smithi016 ceph-mon[29214]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-18T02:56:08.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:08 smithi123 ceph-mon[36358]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-18T02:56:09.866 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:09 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-06-18T02:56:09.867 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:09 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:56:09.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:09 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-06-18T02:56:09.985 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:09 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:56:10.687 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:10 smithi016 ceph-mon[29214]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-18T02:56:10.687 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:10 smithi016 ceph-mon[29214]: Deploying daemon osd.2 on smithi016 2024-06-18T02:56:10.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:10 smithi123 ceph-mon[36358]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-18T02:56:10.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:10 smithi123 ceph-mon[36358]: Deploying daemon osd.2 on smithi016 2024-06-18T02:56:11.886 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:11 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:56:11.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:11 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:56:12.890 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:12 smithi016 ceph-mon[29214]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-18T02:56:12.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:12 smithi123 ceph-mon[36358]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-18T02:56:13.836 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:13 smithi016 ceph-mon[29214]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-18T02:56:13.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:13 smithi123 ceph-mon[36358]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-18T02:56:14.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:14 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:56:14.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:14 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:56:14.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:14 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:56:14.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:14 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:56:14.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:14 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:56:14.985 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:14 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:56:15.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:15 smithi016 ceph-mon[29214]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-18T02:56:15.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:15 smithi123 ceph-mon[36358]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-18T02:56:18.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:56:18.118+0000 7f46217fa640 1 -- 172.21.15.16:0/3646379671 <== mgr.14216 v2:172.21.15.16:6800/3814182425 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f46240f84d0 con 0x7f46140653a0 2024-06-18T02:56:18.119 INFO:teuthology.orchestra.run.smithi016.stdout:Created osd(s) 2 on host 'smithi016' 2024-06-18T02:56:18.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:56:18.120+0000 7f462ade8640 1 -- 172.21.15.16:0/3646379671 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f46140653a0 msgr2=0x7f4614067850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:56:18.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:56:18.120+0000 7f462ade8640 1 --2- 172.21.15.16:0/3646379671 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f46140653a0 0x7f4614067850 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4610004c60 tx=0x7f4610009040 comp rx=0 tx=0).stop 2024-06-18T02:56:18.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:56:18.121+0000 7f462ade8640 1 -- 172.21.15.16:0/3646379671 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4624066470 msgr2=0x7f46240fc8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:56:18.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:56:18.121+0000 7f462ade8640 1 --2- 172.21.15.16:0/3646379671 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4624066470 0x7f46240fc8f0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f4618000c00 tx=0x7f4618009da0 comp rx=0 tx=0).stop 2024-06-18T02:56:18.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:56:18.121+0000 7f462ade8640 1 -- 172.21.15.16:0/3646379671 shutdown_connections 2024-06-18T02:56:18.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:56:18.121+0000 7f462ade8640 1 --2- 172.21.15.16:0/3646379671 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f46140653a0 0x7f4614067850 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:56:18.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:56:18.121+0000 7f462ade8640 1 --2- 172.21.15.16:0/3646379671 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46240ff420 0x7f46240fade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:56:18.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:56:18.121+0000 7f462ade8640 1 --2- 172.21.15.16:0/3646379671 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4624066470 0x7f46240fc8f0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:56:18.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:56:18.121+0000 7f462ade8640 1 -- 172.21.15.16:0/3646379671 >> 172.21.15.16:0/3646379671 conn(0x7f46240f6970 msgr2=0x7f46240f6db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:56:18.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:56:18.121+0000 7f462ade8640 1 -- 172.21.15.16:0/3646379671 shutdown_connections 2024-06-18T02:56:18.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:56:18.121+0000 7f462ade8640 1 -- 172.21.15.16:0/3646379671 wait complete. 2024-06-18T02:56:18.397 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:18 smithi016 ceph-mon[29214]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-18T02:56:18.397 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:18 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:56:18.397 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:18 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:56:18.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:18 smithi123 ceph-mon[36358]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-18T02:56:18.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:18 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:56:18.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:18 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:56:18.749 DEBUG:teuthology.orchestra.run.smithi016:osd.2> sudo journalctl -f -n 0 -u ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968@osd.2.service 2024-06-18T02:56:18.751 INFO:tasks.cephadm:Deploying osd.3 on smithi123 with /dev/vg_nvme/lv_4... 2024-06-18T02:56:18.752 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-06-18T02:56:19.019 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi123/config 2024-06-18T02:56:19.417 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:19 smithi123 ceph-mon[36358]: from='osd.2 [v2:172.21.15.16:6818/1319853338,v1:172.21.15.16:6819/1319853338]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-18T02:56:19.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:19 smithi016 ceph-mon[29214]: from='osd.2 [v2:172.21.15.16:6818/1319853338,v1:172.21.15.16:6819/1319853338]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-18T02:56:20.308 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:20 smithi016 ceph-mon[29214]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-18T02:56:20.308 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:20 smithi016 ceph-mon[29214]: from='osd.2 [v2:172.21.15.16:6818/1319853338,v1:172.21.15.16:6819/1319853338]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-06-18T02:56:20.308 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:20 smithi016 ceph-mon[29214]: osdmap e17: 3 total, 2 up, 3 in 2024-06-18T02:56:20.308 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:20 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-18T02:56:20.308 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:20 smithi016 ceph-mon[29214]: from='osd.2 [v2:172.21.15.16:6818/1319853338,v1:172.21.15.16:6819/1319853338]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-06-18T02:56:20.308 INFO:journalctl@ceph.osd.2.smithi016.stdout:Jun 18 02:56:20 smithi016 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-2[62337]: 2024-06-18T02:56:20.302+0000 7f1b7e71e640 -1 osd.2 0 waiting for initial osdmap 2024-06-18T02:56:20.308 INFO:journalctl@ceph.osd.2.smithi016.stdout:Jun 18 02:56:20 smithi016 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-2[62337]: 2024-06-18T02:56:20.307+0000 7f1b78d54640 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-18T02:56:20.336 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:20 smithi123 ceph-mon[36358]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-18T02:56:20.336 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:20 smithi123 ceph-mon[36358]: from='osd.2 [v2:172.21.15.16:6818/1319853338,v1:172.21.15.16:6819/1319853338]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-06-18T02:56:20.336 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:20 smithi123 ceph-mon[36358]: osdmap e17: 3 total, 2 up, 3 in 2024-06-18T02:56:20.336 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:20 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-18T02:56:20.336 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:20 smithi123 ceph-mon[36358]: from='osd.2 [v2:172.21.15.16:6818/1319853338,v1:172.21.15.16:6819/1319853338]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-06-18T02:56:20.757 INFO:teuthology.orchestra.run.smithi123.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-06-18T02:56:20.846 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-06-18T02:56:20.860 INFO:teuthology.orchestra.run.smithi123.stderr: stderr: 10+0 records in 2024-06-18T02:56:20.860 INFO:teuthology.orchestra.run.smithi123.stderr:10+0 records out 2024-06-18T02:56:20.860 INFO:teuthology.orchestra.run.smithi123.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0126984 s, 826 MB/s 2024-06-18T02:56:20.860 INFO:teuthology.orchestra.run.smithi123.stderr:--> Zapping successful for: 2024-06-18T02:56:21.438 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph orch daemon add osd smithi123:vg_nvme/lv_4 2024-06-18T02:56:21.697 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi123/config 2024-06-18T02:56:21.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:21 smithi123 ceph-mon[36358]: from='osd.2 [v2:172.21.15.16:6818/1319853338,v1:172.21.15.16:6819/1319853338]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-06-18T02:56:21.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:21 smithi123 ceph-mon[36358]: osdmap e18: 3 total, 2 up, 3 in 2024-06-18T02:56:21.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:21 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-18T02:56:21.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:21 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-18T02:56:21.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:21 smithi016 ceph-mon[29214]: from='osd.2 [v2:172.21.15.16:6818/1319853338,v1:172.21.15.16:6819/1319853338]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-06-18T02:56:21.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:21 smithi016 ceph-mon[29214]: osdmap e18: 3 total, 2 up, 3 in 2024-06-18T02:56:21.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:21 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-18T02:56:21.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:21 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-18T02:56:22.484 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:22 smithi123 ceph-mon[36358]: purged_snaps scrub starts 2024-06-18T02:56:22.485 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:22 smithi123 ceph-mon[36358]: purged_snaps scrub ok 2024-06-18T02:56:22.485 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:22 smithi123 ceph-mon[36358]: pgmap v71: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-18T02:56:22.485 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:22 smithi123 ceph-mon[36358]: osd.2 [v2:172.21.15.16:6818/1319853338,v1:172.21.15.16:6819/1319853338] boot 2024-06-18T02:56:22.485 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:22 smithi123 ceph-mon[36358]: osdmap e19: 3 total, 3 up, 3 in 2024-06-18T02:56:22.485 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:22 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-18T02:56:22.485 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:22 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:56:22.485 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:22 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:56:22.509 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:22 smithi016 ceph-mon[29214]: purged_snaps scrub starts 2024-06-18T02:56:22.509 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:22 smithi016 ceph-mon[29214]: purged_snaps scrub ok 2024-06-18T02:56:22.509 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:22 smithi016 ceph-mon[29214]: pgmap v71: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-18T02:56:22.509 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:22 smithi016 ceph-mon[29214]: osd.2 [v2:172.21.15.16:6818/1319853338,v1:172.21.15.16:6819/1319853338] boot 2024-06-18T02:56:22.509 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:22 smithi016 ceph-mon[29214]: osdmap e19: 3 total, 3 up, 3 in 2024-06-18T02:56:22.509 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:22 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-18T02:56:22.510 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:22 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:56:22.510 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:22 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:56:23.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:23.244+0000 7fcdb6b77640 1 -- 172.21.15.123:0/1282748928 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcdb00fb190 msgr2=0x7fcdb00fb5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:56:23.255 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:23.244+0000 7fcdb6b77640 1 --2- 172.21.15.123:0/1282748928 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcdb00fb190 0x7fcdb00fb5a0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fcda40048d0 tx=0x7fcda4009da0 comp rx=0 tx=0).stop 2024-06-18T02:56:23.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:23.244+0000 7fcdb6b77640 1 -- 172.21.15.123:0/1282748928 shutdown_connections 2024-06-18T02:56:23.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:23.244+0000 7fcdb6b77640 1 --2- 172.21.15.123:0/1282748928 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcdb00fc460 0x7fcdb00fc8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:56:23.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:23.244+0000 7fcdb6b77640 1 --2- 172.21.15.123:0/1282748928 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcdb00fb190 0x7fcdb00fb5a0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:56:23.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:23.244+0000 7fcdb6b77640 1 -- 172.21.15.123:0/1282748928 >> 172.21.15.123:0/1282748928 conn(0x7fcdb00f6970 msgr2=0x7fcdb00f8d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:56:23.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:23.244+0000 7fcdb6b77640 1 -- 172.21.15.123:0/1282748928 shutdown_connections 2024-06-18T02:56:23.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:23.244+0000 7fcdb6b77640 1 -- 172.21.15.123:0/1282748928 wait complete. 2024-06-18T02:56:23.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:23.244+0000 7fcdb6b77640 1 Processor -- start 2024-06-18T02:56:23.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:23.244+0000 7fcdb6b77640 1 -- start start 2024-06-18T02:56:23.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:23.245+0000 7fcdb6b77640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcdb00fb190 0x7fcdb010be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:56:23.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:23.245+0000 7fcdb6b77640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcdb00fc460 0x7fcdb010c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:56:23.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:23.245+0000 7fcdb6b77640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb010c9c0 con 0x7fcdb00fb190 2024-06-18T02:56:23.256 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:23.245+0000 7fcdb6b77640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb010d100 con 0x7fcdb00fc460 2024-06-18T02:56:23.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:23.245+0000 7fcdaffff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcdb00fc460 0x7fcdb010c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:56:23.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:23.245+0000 7fcdb4913640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcdb00fb190 0x7fcdb010be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:56:23.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:23.245+0000 7fcdaffff640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcdb00fc460 0x7fcdb010c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:58996/0 (socket says 172.21.15.123:58996) 2024-06-18T02:56:23.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:23.245+0000 7fcdaffff640 1 -- 172.21.15.123:0/1509023120 learned_addr learned my addr 172.21.15.123:0/1509023120 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-06-18T02:56:23.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:23.245+0000 7fcdaffff640 1 -- 172.21.15.123:0/1509023120 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcdb00fb190 msgr2=0x7fcdb010be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:56:23.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:23.245+0000 7fcdaffff640 1 --2- 172.21.15.123:0/1509023120 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcdb00fb190 0x7fcdb010be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:56:23.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:23.245+0000 7fcdaffff640 1 -- 172.21.15.123:0/1509023120 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcda4005040 con 0x7fcdb00fc460 2024-06-18T02:56:23.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:23.245+0000 7fcdb4913640 1 --2- 172.21.15.123:0/1509023120 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcdb00fb190 0x7fcdb010be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-18T02:56:23.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:23.245+0000 7fcdaffff640 1 --2- 172.21.15.123:0/1509023120 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcdb00fc460 0x7fcdb010c340 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c00eb70 tx=0x7fcd9c00b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:56:23.257 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:23.245+0000 7fcdad7fa640 1 -- 172.21.15.123:0/1509023120 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcd9c007c30 con 0x7fcdb00fc460 2024-06-18T02:56:23.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:23.245+0000 7fcdb6b77640 1 -- 172.21.15.123:0/1509023120 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdb010d320 con 0x7fcdb00fc460 2024-06-18T02:56:23.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:23.245+0000 7fcdad7fa640 1 -- 172.21.15.123:0/1509023120 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcd9c00f040 con 0x7fcdb00fc460 2024-06-18T02:56:23.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:23.245+0000 7fcdad7fa640 1 -- 172.21.15.123:0/1509023120 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcd9c0094e0 con 0x7fcdb00fc460 2024-06-18T02:56:23.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:23.245+0000 7fcdb6b77640 1 -- 172.21.15.123:0/1509023120 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdb010d890 con 0x7fcdb00fc460 2024-06-18T02:56:23.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:23.246+0000 7fcdb6b77640 1 -- 172.21.15.123:0/1509023120 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdb00fa5f0 con 0x7fcdb00fc460 2024-06-18T02:56:23.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:23.247+0000 7fcdad7fa640 1 -- 172.21.15.123:0/1509023120 <== mon.1 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 18) v1 ==== 83689+0+0 (secure 0 0 0) 0x7fcd9c009750 con 0x7fcdb00fc460 2024-06-18T02:56:23.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:23.247+0000 7fcdad7fa640 1 --2- 172.21.15.123:0/1509023120 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fcda00653a0 0x7fcda0067850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:56:23.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:23.247+0000 7fcdad7fa640 1 -- 172.21.15.123:0/1509023120 <== mon.1 v2:172.21.15.123:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 2842+0+0 (secure 0 0 0) 0x7fcd9c0818c0 con 0x7fcdb00fc460 2024-06-18T02:56:23.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:23.248+0000 7fcdb4913640 1 --2- 172.21.15.123:0/1509023120 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fcda00653a0 0x7fcda0067850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:56:23.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:23.248+0000 7fcdb4913640 1 --2- 172.21.15.123:0/1509023120 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fcda00653a0 0x7fcda0067850 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fcda4009f80 tx=0x7fcda4035040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:56:23.258 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:23.252+0000 7fcdad7fa640 1 -- 172.21.15.123:0/1509023120 <== mon.1 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fcd9c0506c0 con 0x7fcdb00fc460 2024-06-18T02:56:23.422 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:23.422+0000 7fcdb6b77640 1 -- 172.21.15.123:0/1509023120 --> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fcdb010dbf0 con 0x7fcda00653a0 2024-06-18T02:56:23.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:23 smithi123 ceph-mon[36358]: osdmap e20: 3 total, 3 up, 3 in 2024-06-18T02:56:24.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:23 smithi016 ceph-mon[29214]: osdmap e20: 3 total, 3 up, 3 in 2024-06-18T02:56:24.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:24 smithi016 ceph-mon[29214]: pgmap v74: 0 pgs: ; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-06-18T02:56:24.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:24 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-06-18T02:56:24.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:24 smithi016 ceph-mon[29214]: from='client.24131 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-18T02:56:24.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:24 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-18T02:56:24.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:24 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-18T02:56:24.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:24 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:56:24.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:24 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-06-18T02:56:24.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:24 smithi016 ceph-mon[29214]: osdmap e21: 3 total, 3 up, 3 in 2024-06-18T02:56:24.806 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:24 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-06-18T02:56:24.806 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:24 smithi016 ceph-mon[29214]: Detected new or changed devices on smithi016 2024-06-18T02:56:24.806 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:24 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:56:24.806 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:24 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:56:24.806 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:24 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-18T02:56:24.806 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:24 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-18T02:56:24.806 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:24 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-18T02:56:24.806 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:24 smithi016 ceph-mon[29214]: Adjusting osd_memory_target on smithi016 to 4289M 2024-06-18T02:56:24.806 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:24 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:56:24.806 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:24 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:56:24.806 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:24 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T02:56:24.806 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:24 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:56:24.806 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:24 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:56:24.985 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:24 smithi123 ceph-mon[36358]: pgmap v74: 0 pgs: ; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-06-18T02:56:24.985 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:24 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-06-18T02:56:24.985 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:24 smithi123 ceph-mon[36358]: from='client.24131 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-18T02:56:24.985 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:24 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-18T02:56:24.985 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:24 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-18T02:56:24.985 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:24 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:56:24.985 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:24 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-06-18T02:56:24.985 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:24 smithi123 ceph-mon[36358]: osdmap e21: 3 total, 3 up, 3 in 2024-06-18T02:56:24.985 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:24 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-06-18T02:56:24.985 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:24 smithi123 ceph-mon[36358]: Detected new or changed devices on smithi016 2024-06-18T02:56:24.986 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:24 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:56:24.986 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:24 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:56:24.986 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:24 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-18T02:56:24.986 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:24 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-18T02:56:24.986 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:24 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-18T02:56:24.986 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:24 smithi123 ceph-mon[36358]: Adjusting osd_memory_target on smithi016 to 4289M 2024-06-18T02:56:24.986 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:24 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:56:24.986 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:24 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:56:24.986 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:24 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T02:56:24.986 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:24 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:56:24.986 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:24 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:56:25.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:25 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-06-18T02:56:25.985 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:25 smithi123 ceph-mon[36358]: osdmap e22: 3 total, 3 up, 3 in 2024-06-18T02:56:25.985 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:25 smithi123 ceph-mon[36358]: pgmap v77: 1 pgs: 1 creating+peering; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-06-18T02:56:26.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:25 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-06-18T02:56:26.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:25 smithi016 ceph-mon[29214]: osdmap e22: 3 total, 3 up, 3 in 2024-06-18T02:56:26.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:25 smithi016 ceph-mon[29214]: pgmap v77: 1 pgs: 1 creating+peering; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-06-18T02:56:26.555 INFO:journalctl@ceph.osd.0.smithi016.stdout:Jun 18 02:56:26 smithi016 sudo[66724]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-18T02:56:26.555 INFO:journalctl@ceph.osd.0.smithi016.stdout:Jun 18 02:56:26 smithi016 sudo[66724]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-18T02:56:26.555 INFO:journalctl@ceph.osd.0.smithi016.stdout:Jun 18 02:56:26 smithi016 sudo[66724]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-18T02:56:26.555 INFO:journalctl@ceph.osd.0.smithi016.stdout:Jun 18 02:56:26 smithi016 sudo[66724]: pam_unix(sudo:session): session closed for user root 2024-06-18T02:56:26.908 INFO:journalctl@ceph.osd.0.smithi016.stdout:Jun 18 02:56:26 smithi016 sudo[66772]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-18T02:56:26.908 INFO:journalctl@ceph.osd.0.smithi016.stdout:Jun 18 02:56:26 smithi016 sudo[66772]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-18T02:56:26.908 INFO:journalctl@ceph.osd.0.smithi016.stdout:Jun 18 02:56:26 smithi016 sudo[66772]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-18T02:56:26.908 INFO:journalctl@ceph.osd.0.smithi016.stdout:Jun 18 02:56:26 smithi016 sudo[66772]: pam_unix(sudo:session): session closed for user root 2024-06-18T02:56:26.908 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:26 smithi016 ceph-mon[29214]: osdmap e23: 3 total, 3 up, 3 in 2024-06-18T02:56:26.908 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:26 smithi016 ceph-mon[29214]: from='client.? 172.21.15.123:0/1409803544' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d432cb55-356d-44f1-b6f0-cc98fbec6809"}]: dispatch 2024-06-18T02:56:26.908 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:26 smithi016 ceph-mon[29214]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d432cb55-356d-44f1-b6f0-cc98fbec6809"}]: dispatch 2024-06-18T02:56:26.908 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:26 smithi016 ceph-mon[29214]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d432cb55-356d-44f1-b6f0-cc98fbec6809"}]': finished 2024-06-18T02:56:26.909 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:26 smithi016 ceph-mon[29214]: osdmap e24: 4 total, 3 up, 4 in 2024-06-18T02:56:26.909 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:26 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-18T02:56:26.909 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:26 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:56:26.909 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:26 smithi016 ceph-mon[29214]: from='client.? 172.21.15.123:0/743058208' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-18T02:56:26.909 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:26 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:56:26.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:26 smithi123 ceph-mon[36358]: osdmap e23: 3 total, 3 up, 3 in 2024-06-18T02:56:26.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:26 smithi123 ceph-mon[36358]: from='client.? 172.21.15.123:0/1409803544' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d432cb55-356d-44f1-b6f0-cc98fbec6809"}]: dispatch 2024-06-18T02:56:26.985 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:26 smithi123 ceph-mon[36358]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d432cb55-356d-44f1-b6f0-cc98fbec6809"}]: dispatch 2024-06-18T02:56:26.985 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:26 smithi123 ceph-mon[36358]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d432cb55-356d-44f1-b6f0-cc98fbec6809"}]': finished 2024-06-18T02:56:26.985 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:26 smithi123 ceph-mon[36358]: osdmap e24: 4 total, 3 up, 4 in 2024-06-18T02:56:26.985 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:26 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-18T02:56:26.985 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:26 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:56:26.985 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:26 smithi123 ceph-mon[36358]: from='client.? 172.21.15.123:0/743058208' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-18T02:56:26.985 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:26 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:56:27.305 INFO:journalctl@ceph.osd.1.smithi016.stdout:Jun 18 02:56:27 smithi016 sudo[66802]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-18T02:56:27.305 INFO:journalctl@ceph.osd.1.smithi016.stdout:Jun 18 02:56:27 smithi016 sudo[66802]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-18T02:56:27.305 INFO:journalctl@ceph.osd.1.smithi016.stdout:Jun 18 02:56:27 smithi016 sudo[66802]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-18T02:56:27.305 INFO:journalctl@ceph.osd.1.smithi016.stdout:Jun 18 02:56:27 smithi016 sudo[66802]: pam_unix(sudo:session): session closed for user root 2024-06-18T02:56:27.787 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:27.787+0000 7fcdad7fa640 1 -- 172.21.15.123:0/1509023120 <== mon.1 v2:172.21.15.123:3300/0 7 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fcd9c050a90 con 0x7fcdb00fc460 2024-06-18T02:56:28.054 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:27 smithi016 ceph-mon[29214]: pgmap v80: 1 pgs: 1 creating+peering; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-06-18T02:56:28.055 INFO:journalctl@ceph.osd.1.smithi016.stdout:Jun 18 02:56:27 smithi016 sudo[66868]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-18T02:56:28.055 INFO:journalctl@ceph.osd.1.smithi016.stdout:Jun 18 02:56:27 smithi016 sudo[66868]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-18T02:56:28.055 INFO:journalctl@ceph.osd.1.smithi016.stdout:Jun 18 02:56:27 smithi016 sudo[66868]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-18T02:56:28.055 INFO:journalctl@ceph.osd.1.smithi016.stdout:Jun 18 02:56:27 smithi016 sudo[66868]: pam_unix(sudo:session): session closed for user root 2024-06-18T02:56:28.234 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:27 smithi123 ceph-mon[36358]: pgmap v80: 1 pgs: 1 creating+peering; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-06-18T02:56:28.305 INFO:journalctl@ceph.osd.2.smithi016.stdout:Jun 18 02:56:28 smithi016 sudo[66887]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-18T02:56:28.305 INFO:journalctl@ceph.osd.2.smithi016.stdout:Jun 18 02:56:28 smithi016 sudo[66887]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-18T02:56:28.305 INFO:journalctl@ceph.osd.2.smithi016.stdout:Jun 18 02:56:28 smithi016 sudo[66887]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-18T02:56:28.305 INFO:journalctl@ceph.osd.2.smithi016.stdout:Jun 18 02:56:28 smithi016 sudo[66887]: pam_unix(sudo:session): session closed for user root 2024-06-18T02:56:28.626 INFO:journalctl@ceph.osd.2.smithi016.stdout:Jun 18 02:56:28 smithi016 sudo[66956]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-18T02:56:28.627 INFO:journalctl@ceph.osd.2.smithi016.stdout:Jun 18 02:56:28 smithi016 sudo[66956]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-18T02:56:28.627 INFO:journalctl@ceph.osd.2.smithi016.stdout:Jun 18 02:56:28 smithi016 sudo[66956]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-18T02:56:28.627 INFO:journalctl@ceph.osd.2.smithi016.stdout:Jun 18 02:56:28 smithi016 sudo[66956]: pam_unix(sudo:session): session closed for user root 2024-06-18T02:56:29.305 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:28 smithi016 sudo[67013]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-18T02:56:29.305 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:28 smithi016 sudo[67013]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-18T02:56:29.305 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:28 smithi016 sudo[67013]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-18T02:56:29.305 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:28 smithi016 sudo[67013]: pam_unix(sudo:session): session closed for user root 2024-06-18T02:56:29.708 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:29 smithi123 ceph-mon[36358]: mgrmap e19: smithi016.amxydz(active, since 2m), standbys: smithi123.olpkyi 2024-06-18T02:56:29.708 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:29 smithi123 ceph-mon[36358]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-18T02:56:29.708 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:29 smithi123 sudo[40968]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-18T02:56:29.708 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:29 smithi123 sudo[40968]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-18T02:56:29.708 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:29 smithi123 sudo[40968]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-18T02:56:29.708 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:29 smithi123 sudo[40968]: pam_unix(sudo:session): session closed for user root 2024-06-18T02:56:29.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:29 smithi016 sudo[67018]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-18T02:56:29.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:29 smithi016 sudo[67018]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-18T02:56:29.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:29 smithi016 sudo[67018]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-18T02:56:29.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:29 smithi016 sudo[67018]: pam_unix(sudo:session): session closed for user root 2024-06-18T02:56:29.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:29 smithi016 ceph-mon[29214]: mgrmap e19: smithi016.amxydz(active, since 2m), standbys: smithi123.olpkyi 2024-06-18T02:56:29.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:29 smithi016 ceph-mon[29214]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-18T02:56:29.959 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:29 smithi123 sudo[41065]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-18T02:56:29.959 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:29 smithi123 sudo[41065]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-18T02:56:29.959 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:29 smithi123 sudo[41065]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-18T02:56:29.959 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:29 smithi123 sudo[41065]: pam_unix(sudo:session): session closed for user root 2024-06-18T02:56:30.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:30 smithi123 ceph-mon[36358]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-18T02:56:30.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:30 smithi123 ceph-mon[36358]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-18T02:56:30.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:30 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "mon metadata", "id": "smithi016"}]: dispatch 2024-06-18T02:56:30.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:30 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "mon metadata", "id": "smithi123"}]: dispatch 2024-06-18T02:56:30.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:30 smithi123 ceph-mon[36358]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-18T02:56:30.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:30 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:56:30.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:30 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "mon metadata", "id": "smithi016"}]: dispatch 2024-06-18T02:56:30.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:30 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "mon metadata", "id": "smithi123"}]: dispatch 2024-06-18T02:56:30.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:30 smithi123 ceph-mon[36358]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-18T02:56:30.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:30 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:56:30.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:30 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:56:30.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:30 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:56:30.736 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:30 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T02:56:30.736 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:30 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:56:30.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:30 smithi016 ceph-mon[29214]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-18T02:56:30.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:30 smithi016 ceph-mon[29214]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-18T02:56:30.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:30 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "mon metadata", "id": "smithi016"}]: dispatch 2024-06-18T02:56:30.806 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:30 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "mon metadata", "id": "smithi123"}]: dispatch 2024-06-18T02:56:30.806 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:30 smithi016 ceph-mon[29214]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-18T02:56:30.806 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:30 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:56:30.806 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:30 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "mon metadata", "id": "smithi016"}]: dispatch 2024-06-18T02:56:30.806 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:30 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "mon metadata", "id": "smithi123"}]: dispatch 2024-06-18T02:56:30.806 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:30 smithi016 ceph-mon[29214]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-18T02:56:30.806 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:30 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:56:30.806 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:30 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:56:30.806 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:30 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:56:30.807 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:30 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T02:56:30.807 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:30 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:56:32.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:32 smithi123 ceph-mon[36358]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-18T02:56:32.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:32 smithi016 ceph-mon[29214]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-18T02:56:33.648 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:33 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-06-18T02:56:33.648 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:33 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:56:33.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:33 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-06-18T02:56:33.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:33 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:56:34.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:34 smithi016 ceph-mon[29214]: Deploying daemon osd.3 on smithi123 2024-06-18T02:56:34.556 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:34 smithi016 ceph-mon[29214]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-18T02:56:34.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:34 smithi123 ceph-mon[36358]: Deploying daemon osd.3 on smithi123 2024-06-18T02:56:34.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:34 smithi123 ceph-mon[36358]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-18T02:56:36.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:36 smithi123 ceph-mon[36358]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-06-18T02:56:36.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:36 smithi016 ceph-mon[29214]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-06-18T02:56:38.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:38 smithi123 ceph-mon[36358]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-06-18T02:56:38.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:38 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:56:38.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:38 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:56:38.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:38 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:56:38.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:38 smithi016 ceph-mon[29214]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-06-18T02:56:38.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:38 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:56:38.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:38 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:56:38.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:38 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:56:40.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:40 smithi123 ceph-mon[36358]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-06-18T02:56:40.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:40 smithi016 ceph-mon[29214]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-06-18T02:56:41.804 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:41.803+0000 7fcdad7fa640 1 -- 172.21.15.123:0/1509023120 <== mgr.14216 v2:172.21.15.16:6800/3814182425 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fcdb010dbf0 con 0x7fcda00653a0 2024-06-18T02:56:41.804 INFO:teuthology.orchestra.run.smithi123.stdout:Created osd(s) 3 on host 'smithi123' 2024-06-18T02:56:41.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:41.805+0000 7fcdb6b77640 1 -- 172.21.15.123:0/1509023120 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fcda00653a0 msgr2=0x7fcda0067850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:56:41.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:41.805+0000 7fcdb6b77640 1 --2- 172.21.15.123:0/1509023120 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fcda00653a0 0x7fcda0067850 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fcda4009f80 tx=0x7fcda4035040 comp rx=0 tx=0).stop 2024-06-18T02:56:41.806 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:41.805+0000 7fcdb6b77640 1 -- 172.21.15.123:0/1509023120 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcdb00fc460 msgr2=0x7fcdb010c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:56:41.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:41.805+0000 7fcdb6b77640 1 --2- 172.21.15.123:0/1509023120 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcdb00fc460 0x7fcdb010c340 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c00eb70 tx=0x7fcd9c00b040 comp rx=0 tx=0).stop 2024-06-18T02:56:41.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:41.805+0000 7fcdb6b77640 1 -- 172.21.15.123:0/1509023120 shutdown_connections 2024-06-18T02:56:41.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:41.805+0000 7fcdb6b77640 1 --2- 172.21.15.123:0/1509023120 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fcda00653a0 0x7fcda0067850 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:56:41.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:41.805+0000 7fcdb6b77640 1 --2- 172.21.15.123:0/1509023120 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcdb00fc460 0x7fcdb010c340 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:56:41.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:41.805+0000 7fcdb6b77640 1 --2- 172.21.15.123:0/1509023120 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcdb00fb190 0x7fcdb010be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:56:41.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:41.805+0000 7fcdb6b77640 1 -- 172.21.15.123:0/1509023120 >> 172.21.15.123:0/1509023120 conn(0x7fcdb00f6970 msgr2=0x7fcdb00f8020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:56:41.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:41.805+0000 7fcdb6b77640 1 -- 172.21.15.123:0/1509023120 shutdown_connections 2024-06-18T02:56:41.807 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:41.805+0000 7fcdb6b77640 1 -- 172.21.15.123:0/1509023120 wait complete. 2024-06-18T02:56:41.840 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:41 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:56:41.840 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:41 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:56:41.840 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:41 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:56:41.840 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:41 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T02:56:41.840 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:41 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:56:41.840 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:41 smithi123 ceph-mon[36358]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-06-18T02:56:41.840 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:41 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:56:41.840 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:41 smithi123 ceph-mon[36358]: from='osd.3 [v2:172.21.15.123:6800/2187102998,v1:172.21.15.123:6801/2187102998]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-18T02:56:41.840 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:41 smithi123 ceph-mon[36358]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-18T02:56:42.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:41 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:56:42.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:41 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:56:42.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:41 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:56:42.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:41 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T02:56:42.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:41 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:56:42.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:41 smithi016 ceph-mon[29214]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-06-18T02:56:42.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:41 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:56:42.056 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:41 smithi016 ceph-mon[29214]: from='osd.3 [v2:172.21.15.123:6800/2187102998,v1:172.21.15.123:6801/2187102998]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-18T02:56:42.056 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:41 smithi016 ceph-mon[29214]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-18T02:56:42.563 DEBUG:teuthology.orchestra.run.smithi123:osd.3> sudo journalctl -f -n 0 -u ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968@osd.3.service 2024-06-18T02:56:42.566 INFO:tasks.cephadm:Deploying osd.4 on smithi123 with /dev/vg_nvme/lv_3... 2024-06-18T02:56:42.566 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-06-18T02:56:42.852 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:42 smithi123 ceph-mon[36358]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-06-18T02:56:42.853 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:42 smithi123 ceph-mon[36358]: osdmap e25: 4 total, 3 up, 4 in 2024-06-18T02:56:42.853 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:42 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-18T02:56:42.853 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:42 smithi123 ceph-mon[36358]: from='osd.3 [v2:172.21.15.123:6800/2187102998,v1:172.21.15.123:6801/2187102998]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-06-18T02:56:42.853 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:42 smithi123 ceph-mon[36358]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-06-18T02:56:42.853 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:42 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:56:42.853 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:42 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:56:42.853 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:42 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:56:42.853 INFO:journalctl@ceph.osd.3.smithi123.stdout:Jun 18 02:56:42 smithi123 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-3[41826]: 2024-06-18T02:56:42.730+0000 7fd10c8bb640 -1 osd.3 0 waiting for initial osdmap 2024-06-18T02:56:42.853 INFO:journalctl@ceph.osd.3.smithi123.stdout:Jun 18 02:56:42 smithi123 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-3[41826]: 2024-06-18T02:56:42.735+0000 7fd1076f2640 -1 osd.3 26 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-18T02:56:43.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:42 smithi016 ceph-mon[29214]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-06-18T02:56:43.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:42 smithi016 ceph-mon[29214]: osdmap e25: 4 total, 3 up, 4 in 2024-06-18T02:56:43.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:42 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-18T02:56:43.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:42 smithi016 ceph-mon[29214]: from='osd.3 [v2:172.21.15.123:6800/2187102998,v1:172.21.15.123:6801/2187102998]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-06-18T02:56:43.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:42 smithi016 ceph-mon[29214]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-06-18T02:56:43.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:42 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:56:43.056 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:42 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:56:43.056 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:42 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:56:43.210 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi123/config 2024-06-18T02:56:43.883 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:43 smithi123 ceph-mon[36358]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-06-18T02:56:43.883 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:43 smithi123 ceph-mon[36358]: osdmap e26: 4 total, 3 up, 4 in 2024-06-18T02:56:43.883 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:43 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-18T02:56:43.883 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:43 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-18T02:56:43.883 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:43 smithi123 ceph-mon[36358]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-18T02:56:44.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:43 smithi016 ceph-mon[29214]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-06-18T02:56:44.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:43 smithi016 ceph-mon[29214]: osdmap e26: 4 total, 3 up, 4 in 2024-06-18T02:56:44.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:43 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-18T02:56:44.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:43 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-18T02:56:44.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:43 smithi016 ceph-mon[29214]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-18T02:56:44.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:44 smithi123 ceph-mon[36358]: purged_snaps scrub starts 2024-06-18T02:56:44.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:44 smithi123 ceph-mon[36358]: purged_snaps scrub ok 2024-06-18T02:56:44.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:44 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-18T02:56:44.985 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:44 smithi123 ceph-mon[36358]: osd.3 [v2:172.21.15.123:6800/2187102998,v1:172.21.15.123:6801/2187102998] boot 2024-06-18T02:56:44.985 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:44 smithi123 ceph-mon[36358]: osdmap e27: 4 total, 4 up, 4 in 2024-06-18T02:56:44.985 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:44 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-18T02:56:45.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:44 smithi016 ceph-mon[29214]: purged_snaps scrub starts 2024-06-18T02:56:45.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:44 smithi016 ceph-mon[29214]: purged_snaps scrub ok 2024-06-18T02:56:45.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:44 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-18T02:56:45.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:44 smithi016 ceph-mon[29214]: osd.3 [v2:172.21.15.123:6800/2187102998,v1:172.21.15.123:6801/2187102998] boot 2024-06-18T02:56:45.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:44 smithi016 ceph-mon[29214]: osdmap e27: 4 total, 4 up, 4 in 2024-06-18T02:56:45.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:44 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-18T02:56:45.919 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:45 smithi123 ceph-mon[36358]: osdmap e28: 4 total, 4 up, 4 in 2024-06-18T02:56:45.919 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:45 smithi123 ceph-mon[36358]: pgmap v93: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-18T02:56:46.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:45 smithi016 ceph-mon[29214]: osdmap e28: 4 total, 4 up, 4 in 2024-06-18T02:56:46.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:45 smithi016 ceph-mon[29214]: pgmap v93: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-18T02:56:46.541 INFO:teuthology.orchestra.run.smithi123.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-06-18T02:56:46.643 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-06-18T02:56:46.657 INFO:teuthology.orchestra.run.smithi123.stderr: stderr: 10+0 records in 2024-06-18T02:56:46.658 INFO:teuthology.orchestra.run.smithi123.stderr:10+0 records out 2024-06-18T02:56:46.658 INFO:teuthology.orchestra.run.smithi123.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0125717 s, 834 MB/s 2024-06-18T02:56:46.658 INFO:teuthology.orchestra.run.smithi123.stderr:--> Zapping successful for: 2024-06-18T02:56:46.974 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:46 smithi123 ceph-mon[36358]: osdmap e29: 4 total, 4 up, 4 in 2024-06-18T02:56:46.974 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:46 smithi123 ceph-mon[36358]: Detected new or changed devices on smithi123 2024-06-18T02:56:46.974 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:46 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:56:46.974 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:46 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:56:46.974 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:46 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-18T02:56:46.974 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:46 smithi123 ceph-mon[36358]: Adjusting osd_memory_target on smithi123 to 15936M 2024-06-18T02:56:46.974 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:46 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:56:46.974 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:46 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:56:46.974 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:46 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T02:56:46.975 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:46 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:56:47.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:46 smithi016 ceph-mon[29214]: osdmap e29: 4 total, 4 up, 4 in 2024-06-18T02:56:47.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:46 smithi016 ceph-mon[29214]: Detected new or changed devices on smithi123 2024-06-18T02:56:47.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:46 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:56:47.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:46 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:56:47.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:46 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-18T02:56:47.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:46 smithi016 ceph-mon[29214]: Adjusting osd_memory_target on smithi123 to 15936M 2024-06-18T02:56:47.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:46 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:56:47.056 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:46 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:56:47.056 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:46 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T02:56:47.056 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:46 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:56:47.374 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph orch daemon add osd smithi123:vg_nvme/lv_3 2024-06-18T02:56:47.623 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi123/config 2024-06-18T02:56:48.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:47 smithi016 ceph-mon[29214]: osdmap e30: 4 total, 4 up, 4 in 2024-06-18T02:56:48.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:47 smithi016 ceph-mon[29214]: pgmap v96: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-18T02:56:48.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:47 smithi016 ceph-mon[29214]: osdmap e31: 4 total, 4 up, 4 in 2024-06-18T02:56:48.077 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:47 smithi123 ceph-mon[36358]: osdmap e30: 4 total, 4 up, 4 in 2024-06-18T02:56:48.077 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:47 smithi123 ceph-mon[36358]: pgmap v96: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-18T02:56:48.077 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:47 smithi123 ceph-mon[36358]: osdmap e31: 4 total, 4 up, 4 in 2024-06-18T02:56:49.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:49.280+0000 7fd04475f640 1 -- 172.21.15.123:0/2136549247 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd03c0fc480 msgr2=0x7fd03c0fc8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:56:49.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:49.280+0000 7fd04475f640 1 --2- 172.21.15.123:0/2136549247 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd03c0fc480 0x7fd03c0fc8f0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd0380048d0 tx=0x7fd038009da0 comp rx=0 tx=0).stop 2024-06-18T02:56:49.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:49.280+0000 7fd04475f640 1 -- 172.21.15.123:0/2136549247 shutdown_connections 2024-06-18T02:56:49.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:49.280+0000 7fd04475f640 1 --2- 172.21.15.123:0/2136549247 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd03c0fc480 0x7fd03c0fc8f0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:56:49.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:49.280+0000 7fd04475f640 1 --2- 172.21.15.123:0/2136549247 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd03c0fb1b0 0x7fd03c0fb5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:56:49.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:49.280+0000 7fd04475f640 1 -- 172.21.15.123:0/2136549247 >> 172.21.15.123:0/2136549247 conn(0x7fd03c0f6970 msgr2=0x7fd03c0f8da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:56:49.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:49.280+0000 7fd04475f640 1 -- 172.21.15.123:0/2136549247 shutdown_connections 2024-06-18T02:56:49.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:49.280+0000 7fd04475f640 1 -- 172.21.15.123:0/2136549247 wait complete. 2024-06-18T02:56:49.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:49.281+0000 7fd04475f640 1 Processor -- start 2024-06-18T02:56:49.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:49.281+0000 7fd04475f640 1 -- start start 2024-06-18T02:56:49.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:49.281+0000 7fd04475f640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd03c0fb1b0 0x7fd03c10cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:56:49.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:49.281+0000 7fd04475f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd03c0fc480 0x7fd03c10d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:56:49.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:49.281+0000 7fd04475f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd03c10e350 con 0x7fd03c0fc480 2024-06-18T02:56:49.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:49.281+0000 7fd04475f640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd03c10e4c0 con 0x7fd03c0fb1b0 2024-06-18T02:56:49.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:49.281+0000 7fd0424fb640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd03c0fb1b0 0x7fd03c10cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:56:49.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:49.281+0000 7fd0424fb640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd03c0fb1b0 0x7fd03c10cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:53282/0 (socket says 172.21.15.123:53282) 2024-06-18T02:56:49.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:49.281+0000 7fd0424fb640 1 -- 172.21.15.123:0/4076299061 learned_addr learned my addr 172.21.15.123:0/4076299061 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-06-18T02:56:49.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:49.281+0000 7fd041cfa640 1 --2- 172.21.15.123:0/4076299061 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd03c0fc480 0x7fd03c10d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:56:49.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:49.281+0000 7fd0424fb640 1 -- 172.21.15.123:0/4076299061 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd03c0fc480 msgr2=0x7fd03c10d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:56:49.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:49.281+0000 7fd0424fb640 1 --2- 172.21.15.123:0/4076299061 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd03c0fc480 0x7fd03c10d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:56:49.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:49.281+0000 7fd0424fb640 1 -- 172.21.15.123:0/4076299061 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd038005040 con 0x7fd03c0fb1b0 2024-06-18T02:56:49.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:49.281+0000 7fd0424fb640 1 --2- 172.21.15.123:0/4076299061 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd03c0fb1b0 0x7fd03c10cfa0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fd03000b790 tx=0x7fd0300091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:56:49.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:49.282+0000 7fd02effd640 1 -- 172.21.15.123:0/4076299061 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd03000c700 con 0x7fd03c0fb1b0 2024-06-18T02:56:49.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:49.282+0000 7fd02effd640 1 -- 172.21.15.123:0/4076299061 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd03000cd30 con 0x7fd03c0fb1b0 2024-06-18T02:56:49.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:49.282+0000 7fd04475f640 1 -- 172.21.15.123:0/4076299061 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd03c10e7a0 con 0x7fd03c0fb1b0 2024-06-18T02:56:49.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:49.282+0000 7fd02effd640 1 -- 172.21.15.123:0/4076299061 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd030013370 con 0x7fd03c0fb1b0 2024-06-18T02:56:49.285 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:49.282+0000 7fd04475f640 1 -- 172.21.15.123:0/4076299061 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd03c10ec90 con 0x7fd03c0fb1b0 2024-06-18T02:56:49.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:49.283+0000 7fd04475f640 1 -- 172.21.15.123:0/4076299061 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd03c10dbc0 con 0x7fd03c0fb1b0 2024-06-18T02:56:49.286 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:49.283+0000 7fd02effd640 1 -- 172.21.15.123:0/4076299061 <== mon.1 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fd030012070 con 0x7fd03c0fb1b0 2024-06-18T02:56:49.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:49.283+0000 7fd02effd640 1 --2- 172.21.15.123:0/4076299061 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fd028065430 0x7fd0280678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:56:49.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:49.284+0000 7fd02effd640 1 -- 172.21.15.123:0/4076299061 <== mon.1 v2:172.21.15.123:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 3770+0+0 (secure 0 0 0) 0x7fd0300819d0 con 0x7fd03c0fb1b0 2024-06-18T02:56:49.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:49.284+0000 7fd041cfa640 1 --2- 172.21.15.123:0/4076299061 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fd028065430 0x7fd0280678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:56:49.287 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:49.284+0000 7fd041cfa640 1 --2- 172.21.15.123:0/4076299061 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fd028065430 0x7fd0280678e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd038000c00 tx=0x7fd038035040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:56:49.289 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:49.288+0000 7fd02effd640 1 -- 172.21.15.123:0/4076299061 <== mon.1 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fd030050260 con 0x7fd03c0fb1b0 2024-06-18T02:56:49.463 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:56:49.462+0000 7fd04475f640 1 -- 172.21.15.123:0/4076299061 --> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fd03c0fa6b0 con 0x7fd028065430 2024-06-18T02:56:50.484 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:50 smithi123 ceph-mon[36358]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 128 KiB/s, 0 objects/s recovering 2024-06-18T02:56:50.485 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:50 smithi123 ceph-mon[36358]: from='client.24151 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-18T02:56:50.485 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:50 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-18T02:56:50.485 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:50 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-18T02:56:50.485 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:50 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:56:50.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:50 smithi016 ceph-mon[29214]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 128 KiB/s, 0 objects/s recovering 2024-06-18T02:56:50.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:50 smithi016 ceph-mon[29214]: from='client.24151 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-18T02:56:50.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:50 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-18T02:56:50.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:50 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-18T02:56:50.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:50 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:56:52.484 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:52 smithi123 ceph-mon[36358]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-06-18T02:56:52.484 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:52 smithi123 ceph-mon[36358]: from='client.? 172.21.15.123:0/937780265' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "05cf06af-331b-408e-a7b4-1ae1486c74a0"}]: dispatch 2024-06-18T02:56:52.484 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:52 smithi123 ceph-mon[36358]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "05cf06af-331b-408e-a7b4-1ae1486c74a0"}]: dispatch 2024-06-18T02:56:52.485 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:52 smithi123 ceph-mon[36358]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "05cf06af-331b-408e-a7b4-1ae1486c74a0"}]': finished 2024-06-18T02:56:52.485 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:52 smithi123 ceph-mon[36358]: osdmap e32: 5 total, 4 up, 5 in 2024-06-18T02:56:52.485 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:52 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-18T02:56:52.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:52 smithi016 ceph-mon[29214]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-06-18T02:56:52.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:52 smithi016 ceph-mon[29214]: from='client.? 172.21.15.123:0/937780265' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "05cf06af-331b-408e-a7b4-1ae1486c74a0"}]: dispatch 2024-06-18T02:56:52.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:52 smithi016 ceph-mon[29214]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "05cf06af-331b-408e-a7b4-1ae1486c74a0"}]: dispatch 2024-06-18T02:56:52.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:52 smithi016 ceph-mon[29214]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "05cf06af-331b-408e-a7b4-1ae1486c74a0"}]': finished 2024-06-18T02:56:52.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:52 smithi016 ceph-mon[29214]: osdmap e32: 5 total, 4 up, 5 in 2024-06-18T02:56:52.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:52 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-18T02:56:53.484 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:53 smithi123 ceph-mon[36358]: from='client.? 172.21.15.123:0/3831051494' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-18T02:56:53.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:53 smithi016 ceph-mon[29214]: from='client.? 172.21.15.123:0/3831051494' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-18T02:56:54.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:54 smithi016 ceph-mon[29214]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-06-18T02:56:54.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:54 smithi123 ceph-mon[36358]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-06-18T02:56:56.405 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:56 smithi123 ceph-mon[36358]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-06-18T02:56:56.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:56 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:56:56.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:56 smithi016 ceph-mon[29214]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-06-18T02:56:56.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:56 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:56:58.471 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:58 smithi123 ceph-mon[36358]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-06-18T02:56:58.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:58 smithi016 ceph-mon[29214]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-06-18T02:56:59.484 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:59 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-06-18T02:56:59.485 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:56:59 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:56:59.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:59 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-06-18T02:56:59.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:56:59 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:57:00.728 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:00 smithi123 ceph-mon[36358]: Deploying daemon osd.4 on smithi123 2024-06-18T02:57:00.728 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:00 smithi123 ceph-mon[36358]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-18T02:57:00.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:00 smithi016 ceph-mon[29214]: Deploying daemon osd.4 on smithi123 2024-06-18T02:57:00.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:00 smithi016 ceph-mon[29214]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-18T02:57:02.641 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:02 smithi123 ceph-mon[36358]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-18T02:57:02.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:02 smithi016 ceph-mon[29214]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-18T02:57:04.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:04 smithi016 ceph-mon[29214]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-18T02:57:04.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:04 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:57:04.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:04 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:57:04.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:04 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:57:04.629 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:04 smithi123 ceph-mon[36358]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-18T02:57:04.629 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:04 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:57:04.629 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:04 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:57:04.629 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:04 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:57:06.662 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:06 smithi123 ceph-mon[36358]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-18T02:57:06.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:06 smithi016 ceph-mon[29214]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-18T02:57:07.281 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:07.281+0000 7fd02effd640 1 -- 172.21.15.123:0/4076299061 <== mgr.14216 v2:172.21.15.16:6800/3814182425 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fd03c0fa6b0 con 0x7fd028065430 2024-06-18T02:57:07.281 INFO:teuthology.orchestra.run.smithi123.stdout:Created osd(s) 4 on host 'smithi123' 2024-06-18T02:57:07.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:07.283+0000 7fd04475f640 1 -- 172.21.15.123:0/4076299061 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fd028065430 msgr2=0x7fd0280678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:57:07.283 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:07.283+0000 7fd04475f640 1 --2- 172.21.15.123:0/4076299061 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fd028065430 0x7fd0280678e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd038000c00 tx=0x7fd038035040 comp rx=0 tx=0).stop 2024-06-18T02:57:07.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:07.283+0000 7fd04475f640 1 -- 172.21.15.123:0/4076299061 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd03c0fb1b0 msgr2=0x7fd03c10cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:57:07.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:07.283+0000 7fd04475f640 1 --2- 172.21.15.123:0/4076299061 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd03c0fb1b0 0x7fd03c10cfa0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fd03000b790 tx=0x7fd0300091d0 comp rx=0 tx=0).stop 2024-06-18T02:57:07.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:07.283+0000 7fd04475f640 1 -- 172.21.15.123:0/4076299061 shutdown_connections 2024-06-18T02:57:07.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:07.283+0000 7fd04475f640 1 --2- 172.21.15.123:0/4076299061 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fd028065430 0x7fd0280678e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:57:07.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:07.283+0000 7fd04475f640 1 --2- 172.21.15.123:0/4076299061 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd03c0fc480 0x7fd03c10d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:57:07.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:07.283+0000 7fd04475f640 1 --2- 172.21.15.123:0/4076299061 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd03c0fb1b0 0x7fd03c10cfa0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:57:07.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:07.283+0000 7fd04475f640 1 -- 172.21.15.123:0/4076299061 >> 172.21.15.123:0/4076299061 conn(0x7fd03c0f6970 msgr2=0x7fd03c065d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:57:07.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:07.283+0000 7fd04475f640 1 -- 172.21.15.123:0/4076299061 shutdown_connections 2024-06-18T02:57:07.284 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:07.283+0000 7fd04475f640 1 -- 172.21.15.123:0/4076299061 wait complete. 2024-06-18T02:57:08.186 DEBUG:teuthology.orchestra.run.smithi123:osd.4> sudo journalctl -f -n 0 -u ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968@osd.4.service 2024-06-18T02:57:08.189 INFO:tasks.cephadm:Deploying osd.5 on smithi123 with /dev/vg_nvme/lv_2... 2024-06-18T02:57:08.189 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-06-18T02:57:08.484 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:08 smithi123 ceph-mon[36358]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-18T02:57:08.485 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:08 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:57:08.485 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:08 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:57:08.485 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:08 smithi123 ceph-mon[36358]: from='osd.4 [v2:172.21.15.123:6808/4038205871,v1:172.21.15.123:6809/4038205871]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-18T02:57:08.485 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:08 smithi123 ceph-mon[36358]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-18T02:57:08.503 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi123/config 2024-06-18T02:57:08.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:08 smithi016 ceph-mon[29214]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-18T02:57:08.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:08 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:57:08.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:08 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:57:08.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:08 smithi016 ceph-mon[29214]: from='osd.4 [v2:172.21.15.123:6808/4038205871,v1:172.21.15.123:6809/4038205871]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-18T02:57:08.556 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:08 smithi016 ceph-mon[29214]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-18T02:57:09.454 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:09 smithi123 ceph-mon[36358]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-06-18T02:57:09.454 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:09 smithi123 ceph-mon[36358]: osdmap e33: 5 total, 4 up, 5 in 2024-06-18T02:57:09.454 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:09 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-18T02:57:09.454 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:09 smithi123 ceph-mon[36358]: from='osd.4 [v2:172.21.15.123:6808/4038205871,v1:172.21.15.123:6809/4038205871]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-06-18T02:57:09.454 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:09 smithi123 ceph-mon[36358]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-06-18T02:57:09.454 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:09 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:57:09.454 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:09 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:57:09.455 INFO:journalctl@ceph.osd.4.smithi123.stdout:Jun 18 02:57:09 smithi123 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-4[48669]: 2024-06-18T02:57:09.343+0000 7fb82400e640 -1 osd.4 0 waiting for initial osdmap 2024-06-18T02:57:09.455 INFO:journalctl@ceph.osd.4.smithi123.stdout:Jun 18 02:57:09 smithi123 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-4[48669]: 2024-06-18T02:57:09.350+0000 7fb81ee45640 -1 osd.4 34 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-18T02:57:09.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:09 smithi016 ceph-mon[29214]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-06-18T02:57:09.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:09 smithi016 ceph-mon[29214]: osdmap e33: 5 total, 4 up, 5 in 2024-06-18T02:57:09.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:09 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-18T02:57:09.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:09 smithi016 ceph-mon[29214]: from='osd.4 [v2:172.21.15.123:6808/4038205871,v1:172.21.15.123:6809/4038205871]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-06-18T02:57:09.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:09 smithi016 ceph-mon[29214]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-06-18T02:57:09.806 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:09 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:57:09.806 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:09 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:57:10.500 INFO:teuthology.orchestra.run.smithi123.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-06-18T02:57:10.602 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-06-18T02:57:10.602 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:10 smithi123 ceph-mon[36358]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-18T02:57:10.602 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:10 smithi123 ceph-mon[36358]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-06-18T02:57:10.603 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:10 smithi123 ceph-mon[36358]: osdmap e34: 5 total, 4 up, 5 in 2024-06-18T02:57:10.603 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:10 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-18T02:57:10.603 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:10 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-18T02:57:10.617 INFO:teuthology.orchestra.run.smithi123.stderr: stderr: 10+0 records in 2024-06-18T02:57:10.617 INFO:teuthology.orchestra.run.smithi123.stderr:10+0 records out 2024-06-18T02:57:10.617 INFO:teuthology.orchestra.run.smithi123.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0127779 s, 821 MB/s 2024-06-18T02:57:10.617 INFO:teuthology.orchestra.run.smithi123.stderr:--> Zapping successful for: 2024-06-18T02:57:10.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:10 smithi016 ceph-mon[29214]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-18T02:57:10.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:10 smithi016 ceph-mon[29214]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-06-18T02:57:10.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:10 smithi016 ceph-mon[29214]: osdmap e34: 5 total, 4 up, 5 in 2024-06-18T02:57:10.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:10 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-18T02:57:10.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:10 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-18T02:57:11.254 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph orch daemon add osd smithi123:vg_nvme/lv_2 2024-06-18T02:57:11.484 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:11 smithi123 ceph-mon[36358]: purged_snaps scrub starts 2024-06-18T02:57:11.485 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:11 smithi123 ceph-mon[36358]: purged_snaps scrub ok 2024-06-18T02:57:11.485 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:11 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-18T02:57:11.485 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:11 smithi123 ceph-mon[36358]: osd.4 [v2:172.21.15.123:6808/4038205871,v1:172.21.15.123:6809/4038205871] boot 2024-06-18T02:57:11.485 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:11 smithi123 ceph-mon[36358]: osdmap e35: 5 total, 5 up, 5 in 2024-06-18T02:57:11.485 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:11 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-18T02:57:11.485 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:11 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:57:11.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:11 smithi016 ceph-mon[29214]: purged_snaps scrub starts 2024-06-18T02:57:11.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:11 smithi016 ceph-mon[29214]: purged_snaps scrub ok 2024-06-18T02:57:11.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:11 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-18T02:57:11.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:11 smithi016 ceph-mon[29214]: osd.4 [v2:172.21.15.123:6808/4038205871,v1:172.21.15.123:6809/4038205871] boot 2024-06-18T02:57:11.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:11 smithi016 ceph-mon[29214]: osdmap e35: 5 total, 5 up, 5 in 2024-06-18T02:57:11.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:11 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-18T02:57:11.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:11 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:57:12.077 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi123/config 2024-06-18T02:57:12.484 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:12 smithi123 ceph-mon[36358]: pgmap v113: 1 pgs: 1 peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-18T02:57:12.484 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:12 smithi123 ceph-mon[36358]: osdmap e36: 5 total, 5 up, 5 in 2024-06-18T02:57:12.706 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:12 smithi016 ceph-mon[29214]: pgmap v113: 1 pgs: 1 peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-18T02:57:12.707 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:12 smithi016 ceph-mon[29214]: osdmap e36: 5 total, 5 up, 5 in 2024-06-18T02:57:14.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:14.198+0000 7fdf6d8df640 1 -- 172.21.15.123:0/2540372966 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdf680fb1b0 msgr2=0x7fdf680fb5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:57:14.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:14.198+0000 7fdf6d8df640 1 --2- 172.21.15.123:0/2540372966 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdf680fb1b0 0x7fdf680fb5c0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fdf580048d0 tx=0x7fdf58009da0 comp rx=0 tx=0).stop 2024-06-18T02:57:14.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:14.198+0000 7fdf6d8df640 1 -- 172.21.15.123:0/2540372966 shutdown_connections 2024-06-18T02:57:14.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:14.198+0000 7fdf6d8df640 1 --2- 172.21.15.123:0/2540372966 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf680fc480 0x7fdf680fc8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:57:14.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:14.198+0000 7fdf6d8df640 1 --2- 172.21.15.123:0/2540372966 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdf680fb1b0 0x7fdf680fb5c0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:57:14.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:14.198+0000 7fdf6d8df640 1 -- 172.21.15.123:0/2540372966 >> 172.21.15.123:0/2540372966 conn(0x7fdf680f6970 msgr2=0x7fdf680f8da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:57:14.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:14.198+0000 7fdf6d8df640 1 -- 172.21.15.123:0/2540372966 shutdown_connections 2024-06-18T02:57:14.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:14.198+0000 7fdf6d8df640 1 -- 172.21.15.123:0/2540372966 wait complete. 2024-06-18T02:57:14.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:14.199+0000 7fdf6d8df640 1 Processor -- start 2024-06-18T02:57:14.227 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:14.199+0000 7fdf6d8df640 1 -- start start 2024-06-18T02:57:14.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:14.199+0000 7fdf6d8df640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdf680fb1b0 0x7fdf680ff280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:57:14.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:14.199+0000 7fdf6d8df640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf680fc480 0x7fdf680fd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:57:14.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:14.199+0000 7fdf6d8df640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf680fdd00 con 0x7fdf680fc480 2024-06-18T02:57:14.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:14.199+0000 7fdf6d8df640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf680fde40 con 0x7fdf680fb1b0 2024-06-18T02:57:14.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:14.199+0000 7fdf66ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdf680fb1b0 0x7fdf680ff280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:57:14.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:14.199+0000 7fdf66ffd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdf680fb1b0 0x7fdf680ff280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49012/0 (socket says 172.21.15.123:49012) 2024-06-18T02:57:14.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:14.199+0000 7fdf66ffd640 1 -- 172.21.15.123:0/2058946229 learned_addr learned my addr 172.21.15.123:0/2058946229 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-06-18T02:57:14.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:14.199+0000 7fdf5e7fc640 1 --2- 172.21.15.123:0/2058946229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf680fc480 0x7fdf680fd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:57:14.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:14.199+0000 7fdf66ffd640 1 -- 172.21.15.123:0/2058946229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf680fc480 msgr2=0x7fdf680fd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:57:14.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:14.199+0000 7fdf66ffd640 1 --2- 172.21.15.123:0/2058946229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf680fc480 0x7fdf680fd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:57:14.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:14.199+0000 7fdf66ffd640 1 -- 172.21.15.123:0/2058946229 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf58005040 con 0x7fdf680fb1b0 2024-06-18T02:57:14.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:14.199+0000 7fdf66ffd640 1 --2- 172.21.15.123:0/2058946229 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdf680fb1b0 0x7fdf680ff280 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fdf58009f80 tx=0x7fdf58009da0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:57:14.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:14.200+0000 7fdf5ffff640 1 -- 172.21.15.123:0/2058946229 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdf58038070 con 0x7fdf680fb1b0 2024-06-18T02:57:14.228 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:14.200+0000 7fdf5ffff640 1 -- 172.21.15.123:0/2058946229 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdf58032970 con 0x7fdf680fb1b0 2024-06-18T02:57:14.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:14.200+0000 7fdf6d8df640 1 -- 172.21.15.123:0/2058946229 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf680fe0c0 con 0x7fdf680fb1b0 2024-06-18T02:57:14.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:14.200+0000 7fdf5ffff640 1 -- 172.21.15.123:0/2058946229 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdf58033860 con 0x7fdf680fb1b0 2024-06-18T02:57:14.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:14.200+0000 7fdf6d8df640 1 -- 172.21.15.123:0/2058946229 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf680fe600 con 0x7fdf680fb1b0 2024-06-18T02:57:14.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:14.201+0000 7fdf6d8df640 1 -- 172.21.15.123:0/2058946229 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf680fa610 con 0x7fdf680fb1b0 2024-06-18T02:57:14.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:14.201+0000 7fdf5ffff640 1 -- 172.21.15.123:0/2058946229 <== mon.1 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fdf58030020 con 0x7fdf680fb1b0 2024-06-18T02:57:14.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:14.202+0000 7fdf5ffff640 1 --2- 172.21.15.123:0/2058946229 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fdf54065430 0x7fdf540678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:57:14.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:14.202+0000 7fdf5ffff640 1 -- 172.21.15.123:0/2058946229 <== mon.1 v2:172.21.15.123:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4202+0+0 (secure 0 0 0) 0x7fdf580a2190 con 0x7fdf680fb1b0 2024-06-18T02:57:14.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:14.202+0000 7fdf5e7fc640 1 --2- 172.21.15.123:0/2058946229 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fdf54065430 0x7fdf540678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:57:14.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:14.202+0000 7fdf5e7fc640 1 --2- 172.21.15.123:0/2058946229 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fdf54065430 0x7fdf540678e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fdf50004ed0 tx=0x7fdf5000b040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:57:14.229 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:14.206+0000 7fdf5ffff640 1 -- 172.21.15.123:0/2058946229 <== mon.1 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fdf580708a0 con 0x7fdf680fb1b0 2024-06-18T02:57:14.379 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:14.379+0000 7fdf6d8df640 1 -- 172.21.15.123:0/2058946229 --> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fdf6805d8b0 con 0x7fdf54065430 2024-06-18T02:57:14.558 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:14 smithi016 ceph-mon[29214]: pgmap v115: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-18T02:57:14.558 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:14 smithi016 ceph-mon[29214]: Detected new or changed devices on smithi123 2024-06-18T02:57:14.558 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:14 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:57:14.558 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:14 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:57:14.558 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:14 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-18T02:57:14.558 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:14 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-18T02:57:14.558 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:14 smithi016 ceph-mon[29214]: Adjusting osd_memory_target on smithi123 to 7968M 2024-06-18T02:57:14.558 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:14 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:57:14.559 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:14 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:57:14.559 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:14 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T02:57:14.559 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:14 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:57:14.559 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:14 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:57:14.559 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:14 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:57:14.559 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:14 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T02:57:14.559 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:14 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:57:14.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:14 smithi123 ceph-mon[36358]: pgmap v115: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-18T02:57:14.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:14 smithi123 ceph-mon[36358]: Detected new or changed devices on smithi123 2024-06-18T02:57:14.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:14 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:57:14.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:14 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:57:14.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:14 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-18T02:57:14.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:14 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-18T02:57:14.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:14 smithi123 ceph-mon[36358]: Adjusting osd_memory_target on smithi123 to 7968M 2024-06-18T02:57:14.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:14 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:57:14.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:14 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:57:14.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:14 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T02:57:14.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:14 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:57:14.736 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:14 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:57:14.736 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:14 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:57:14.736 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:14 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T02:57:14.736 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:14 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:57:15.708 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:15 smithi123 ceph-mon[36358]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-18T02:57:15.708 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:15 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-18T02:57:15.709 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:15 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-18T02:57:15.709 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:15 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:57:15.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:15 smithi016 ceph-mon[29214]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi123:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-18T02:57:15.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:15 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-18T02:57:15.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:15 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-18T02:57:15.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:15 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:57:16.699 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:16 smithi123 ceph-mon[36358]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-18T02:57:16.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:16 smithi016 ceph-mon[29214]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-18T02:57:17.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:17 smithi123 ceph-mon[36358]: from='client.? 172.21.15.123:0/908428062' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "faa63f45-ee41-485f-afad-9762cbee8f82"}]: dispatch 2024-06-18T02:57:17.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:17 smithi123 ceph-mon[36358]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "faa63f45-ee41-485f-afad-9762cbee8f82"}]: dispatch 2024-06-18T02:57:17.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:17 smithi123 ceph-mon[36358]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "faa63f45-ee41-485f-afad-9762cbee8f82"}]': finished 2024-06-18T02:57:17.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:17 smithi123 ceph-mon[36358]: osdmap e37: 6 total, 5 up, 6 in 2024-06-18T02:57:17.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:17 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-18T02:57:17.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:17 smithi123 ceph-mon[36358]: from='client.? 172.21.15.123:0/1755864354' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-18T02:57:17.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:17 smithi016 ceph-mon[29214]: from='client.? 172.21.15.123:0/908428062' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "faa63f45-ee41-485f-afad-9762cbee8f82"}]: dispatch 2024-06-18T02:57:17.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:17 smithi016 ceph-mon[29214]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "faa63f45-ee41-485f-afad-9762cbee8f82"}]: dispatch 2024-06-18T02:57:17.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:17 smithi016 ceph-mon[29214]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "faa63f45-ee41-485f-afad-9762cbee8f82"}]': finished 2024-06-18T02:57:17.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:17 smithi016 ceph-mon[29214]: osdmap e37: 6 total, 5 up, 6 in 2024-06-18T02:57:17.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:17 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-18T02:57:17.806 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:17 smithi016 ceph-mon[29214]: from='client.? 172.21.15.123:0/1755864354' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-18T02:57:18.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:18 smithi123 ceph-mon[36358]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-18T02:57:18.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:18 smithi016 ceph-mon[29214]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-18T02:57:20.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:20 smithi123 ceph-mon[36358]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-18T02:57:20.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:20 smithi016 ceph-mon[29214]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-18T02:57:22.637 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:22 smithi123 ceph-mon[36358]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-18T02:57:22.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:22 smithi016 ceph-mon[29214]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-18T02:57:24.675 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:24 smithi016 ceph-mon[29214]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-18T02:57:24.675 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:24 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-06-18T02:57:24.675 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:24 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:57:24.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:24 smithi123 ceph-mon[36358]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-18T02:57:24.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:24 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-06-18T02:57:24.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:24 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:57:25.691 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:25 smithi123 ceph-mon[36358]: Deploying daemon osd.5 on smithi123 2024-06-18T02:57:25.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:25 smithi016 ceph-mon[29214]: Deploying daemon osd.5 on smithi123 2024-06-18T02:57:26.669 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:26 smithi123 ceph-mon[36358]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-18T02:57:26.669 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:26 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:57:26.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:26 smithi016 ceph-mon[29214]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-18T02:57:26.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:26 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:57:28.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:28 smithi123 ceph-mon[36358]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-18T02:57:28.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:28 smithi016 ceph-mon[29214]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-18T02:57:29.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:29 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:57:29.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:29 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:57:29.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:29 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:57:29.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:29 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:57:29.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:29 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:57:29.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:29 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:57:30.660 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:30 smithi123 ceph-mon[36358]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-18T02:57:30.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:30 smithi016 ceph-mon[29214]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-18T02:57:32.711 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:32 smithi123 ceph-mon[36358]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-18T02:57:32.712 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:32 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:57:32.712 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:32 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:57:32.747 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:32.747+0000 7fdf5ffff640 1 -- 172.21.15.123:0/2058946229 <== mgr.14216 v2:172.21.15.16:6800/3814182425 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fdf6805d8b0 con 0x7fdf54065430 2024-06-18T02:57:32.747 INFO:teuthology.orchestra.run.smithi123.stdout:Created osd(s) 5 on host 'smithi123' 2024-06-18T02:57:32.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:32.749+0000 7fdf6d8df640 1 -- 172.21.15.123:0/2058946229 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fdf54065430 msgr2=0x7fdf540678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:57:32.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:32.749+0000 7fdf6d8df640 1 --2- 172.21.15.123:0/2058946229 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fdf54065430 0x7fdf540678e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fdf50004ed0 tx=0x7fdf5000b040 comp rx=0 tx=0).stop 2024-06-18T02:57:32.749 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:32.749+0000 7fdf6d8df640 1 -- 172.21.15.123:0/2058946229 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdf680fb1b0 msgr2=0x7fdf680ff280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:57:32.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:32.749+0000 7fdf6d8df640 1 --2- 172.21.15.123:0/2058946229 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdf680fb1b0 0x7fdf680ff280 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fdf58009f80 tx=0x7fdf58009da0 comp rx=0 tx=0).stop 2024-06-18T02:57:32.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:32.749+0000 7fdf6d8df640 1 -- 172.21.15.123:0/2058946229 shutdown_connections 2024-06-18T02:57:32.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:32.749+0000 7fdf6d8df640 1 --2- 172.21.15.123:0/2058946229 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fdf54065430 0x7fdf540678e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:57:32.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:32.749+0000 7fdf6d8df640 1 --2- 172.21.15.123:0/2058946229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf680fc480 0x7fdf680fd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:57:32.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:32.749+0000 7fdf6d8df640 1 --2- 172.21.15.123:0/2058946229 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdf680fb1b0 0x7fdf680ff280 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:57:32.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:32.749+0000 7fdf6d8df640 1 -- 172.21.15.123:0/2058946229 >> 172.21.15.123:0/2058946229 conn(0x7fdf680f6970 msgr2=0x7fdf68065da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:57:32.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:32.750+0000 7fdf6d8df640 1 -- 172.21.15.123:0/2058946229 shutdown_connections 2024-06-18T02:57:32.750 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:32.750+0000 7fdf6d8df640 1 -- 172.21.15.123:0/2058946229 wait complete. 2024-06-18T02:57:32.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:32 smithi016 ceph-mon[29214]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-18T02:57:32.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:32 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:57:32.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:32 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:57:33.288 DEBUG:teuthology.orchestra.run.smithi123:osd.5> sudo journalctl -f -n 0 -u ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968@osd.5.service 2024-06-18T02:57:33.290 INFO:tasks.cephadm:Waiting for 6 OSDs to come up... 2024-06-18T02:57:33.291 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph osd stat -f json 2024-06-18T02:57:33.556 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T02:57:33.659 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:33 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:57:33.660 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:33 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T02:57:33.660 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:33 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:57:33.660 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:33 smithi123 ceph-mon[36358]: from='osd.5 [v2:172.21.15.123:6816/4116323333,v1:172.21.15.123:6817/4116323333]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-18T02:57:33.660 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:33 smithi123 ceph-mon[36358]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-18T02:57:33.660 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:33 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:57:33.660 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:33 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:57:33.660 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:33 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:57:33.676 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:33 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:57:33.677 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:33 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T02:57:33.677 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:33 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:57:33.677 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:33 smithi016 ceph-mon[29214]: from='osd.5 [v2:172.21.15.123:6816/4116323333,v1:172.21.15.123:6817/4116323333]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-18T02:57:33.677 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:33 smithi016 ceph-mon[29214]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-18T02:57:33.677 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:33 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:57:33.677 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:33 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:57:33.677 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:33 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:57:34.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:34 smithi016 ceph-mon[29214]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-18T02:57:34.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:34 smithi016 ceph-mon[29214]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-06-18T02:57:34.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:34 smithi016 ceph-mon[29214]: osdmap e38: 6 total, 5 up, 6 in 2024-06-18T02:57:34.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:34 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-18T02:57:34.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:34 smithi016 ceph-mon[29214]: from='osd.5 [v2:172.21.15.123:6816/4116323333,v1:172.21.15.123:6817/4116323333]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-06-18T02:57:34.556 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:34 smithi016 ceph-mon[29214]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-06-18T02:57:34.677 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:34 smithi123 ceph-mon[36358]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-18T02:57:34.677 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:34 smithi123 ceph-mon[36358]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-06-18T02:57:34.677 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:34 smithi123 ceph-mon[36358]: osdmap e38: 6 total, 5 up, 6 in 2024-06-18T02:57:34.678 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:34 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-18T02:57:34.678 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:34 smithi123 ceph-mon[36358]: from='osd.5 [v2:172.21.15.123:6816/4116323333,v1:172.21.15.123:6817/4116323333]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-06-18T02:57:34.678 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:34 smithi123 ceph-mon[36358]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]: dispatch 2024-06-18T02:57:34.678 INFO:journalctl@ceph.osd.5.smithi123.stdout:Jun 18 02:57:34 smithi123 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-5[55214]: 2024-06-18T02:57:34.448+0000 7f954e108640 -1 osd.5 0 waiting for initial osdmap 2024-06-18T02:57:34.678 INFO:journalctl@ceph.osd.5.smithi123.stdout:Jun 18 02:57:34 smithi123 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-5[55214]: 2024-06-18T02:57:34.455+0000 7f9546f3b640 -1 osd.5 39 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-18T02:57:35.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:35.220+0000 7f0cd10aa640 1 -- 172.21.15.16:0/4063243243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ccc066410 msgr2=0x7f0ccc0feeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:57:35.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:35.220+0000 7f0cd10aa640 1 --2- 172.21.15.16:0/4063243243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ccc066410 0x7f0ccc0feeb0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc0048d0 tx=0x7f0cbc009da0 comp rx=0 tx=0).stop 2024-06-18T02:57:35.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:35.220+0000 7f0cd10aa640 1 -- 172.21.15.16:0/4063243243 shutdown_connections 2024-06-18T02:57:35.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:35.220+0000 7f0cd10aa640 1 --2- 172.21.15.16:0/4063243243 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ccc0ff3f0 0x7f0ccc1017d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:57:35.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:35.220+0000 7f0cd10aa640 1 --2- 172.21.15.16:0/4063243243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ccc066410 0x7f0ccc0feeb0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:57:35.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:35.220+0000 7f0cd10aa640 1 -- 172.21.15.16:0/4063243243 >> 172.21.15.16:0/4063243243 conn(0x7f0ccc0f6980 msgr2=0x7f0ccc0f8d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:57:35.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:35.220+0000 7f0cd10aa640 1 -- 172.21.15.16:0/4063243243 shutdown_connections 2024-06-18T02:57:35.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:35.220+0000 7f0cd10aa640 1 -- 172.21.15.16:0/4063243243 wait complete. 2024-06-18T02:57:35.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:35.221+0000 7f0cd10aa640 1 Processor -- start 2024-06-18T02:57:35.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:35.221+0000 7f0cd10aa640 1 -- start start 2024-06-18T02:57:35.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:35.221+0000 7f0cd10aa640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ccc066410 0x7f0ccc0fc8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:57:35.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:35.221+0000 7f0cd10aa640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ccc0ff3f0 0x7f0ccc0fadb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:57:35.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:35.221+0000 7f0cd10aa640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ccc0fb340 con 0x7f0ccc066410 2024-06-18T02:57:35.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:35.221+0000 7f0cd10aa640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ccc0fb480 con 0x7f0ccc0ff3f0 2024-06-18T02:57:35.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:35.221+0000 7f0ccad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ccc066410 0x7f0ccc0fc8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:57:35.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:35.221+0000 7f0ccad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ccc066410 0x7f0ccc0fc8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48232/0 (socket says 172.21.15.16:48232) 2024-06-18T02:57:35.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:35.221+0000 7f0ccad9d640 1 -- 172.21.15.16:0/3686247648 learned_addr learned my addr 172.21.15.16:0/3686247648 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:57:35.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:35.221+0000 7f0cca59c640 1 --2- 172.21.15.16:0/3686247648 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ccc0ff3f0 0x7f0ccc0fadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:57:35.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:35.221+0000 7f0ccad9d640 1 -- 172.21.15.16:0/3686247648 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ccc0ff3f0 msgr2=0x7f0ccc0fadb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:57:35.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:35.221+0000 7f0ccad9d640 1 --2- 172.21.15.16:0/3686247648 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ccc0ff3f0 0x7f0ccc0fadb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:57:35.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:35.221+0000 7f0ccad9d640 1 -- 172.21.15.16:0/3686247648 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cbc005040 con 0x7f0ccc066410 2024-06-18T02:57:35.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:35.221+0000 7f0ccad9d640 1 --2- 172.21.15.16:0/3686247648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ccc066410 0x7f0ccc0fc8c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc000c00 tx=0x7f0cbc009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:57:35.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:35.222+0000 7f0cb37fe640 1 -- 172.21.15.16:0/3686247648 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0cbc038070 con 0x7f0ccc066410 2024-06-18T02:57:35.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:35.222+0000 7f0cd10aa640 1 -- 172.21.15.16:0/3686247648 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ccc0fb700 con 0x7f0ccc066410 2024-06-18T02:57:35.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:35.222+0000 7f0cb37fe640 1 -- 172.21.15.16:0/3686247648 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0cbc0092b0 con 0x7f0ccc066410 2024-06-18T02:57:35.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:35.222+0000 7f0cb37fe640 1 -- 172.21.15.16:0/3686247648 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0cbc033780 con 0x7f0ccc066410 2024-06-18T02:57:35.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:35.222+0000 7f0cd10aa640 1 -- 172.21.15.16:0/3686247648 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ccc114fc0 con 0x7f0ccc066410 2024-06-18T02:57:35.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:35.223+0000 7f0cd10aa640 1 -- 172.21.15.16:0/3686247648 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ccc0fa600 con 0x7f0ccc066410 2024-06-18T02:57:35.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:35.223+0000 7f0cb37fe640 1 -- 172.21.15.16:0/3686247648 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f0cbc030020 con 0x7f0ccc066410 2024-06-18T02:57:35.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:35.223+0000 7f0cb37fe640 1 --2- 172.21.15.16:0/3686247648 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f0cb4065430 0x7f0cb40678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:57:35.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:35.224+0000 7f0cb37fe640 1 -- 172.21.15.16:0/3686247648 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4354+0+0 (secure 0 0 0) 0x7f0cbc0a20a0 con 0x7f0ccc066410 2024-06-18T02:57:35.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:35.224+0000 7f0cca59c640 1 --2- 172.21.15.16:0/3686247648 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f0cb4065430 0x7f0cb40678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:57:35.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:35.224+0000 7f0cca59c640 1 --2- 172.21.15.16:0/3686247648 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f0cb4065430 0x7f0cb40678e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0cc0009af0 tx=0x7f0cc000b040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:57:35.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:35.228+0000 7f0cb37fe640 1 -- 172.21.15.16:0/3686247648 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f0cbc070710 con 0x7f0ccc066410 2024-06-18T02:57:35.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:35.406+0000 7f0cd10aa640 1 -- 172.21.15.16:0/3686247648 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f0ccc04b480 con 0x7f0ccc066410 2024-06-18T02:57:35.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:35.406+0000 7f0cb37fe640 1 -- 172.21.15.16:0/3686247648 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v39) v1 ==== 74+0+130 (secure 0 0 0) 0x7f0cbc073d30 con 0x7f0ccc066410 2024-06-18T02:57:35.407 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:57:35.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:35.408+0000 7f0cd10aa640 1 -- 172.21.15.16:0/3686247648 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f0cb4065430 msgr2=0x7f0cb40678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:57:35.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:35.408+0000 7f0cd10aa640 1 --2- 172.21.15.16:0/3686247648 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f0cb4065430 0x7f0cb40678e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0cc0009af0 tx=0x7f0cc000b040 comp rx=0 tx=0).stop 2024-06-18T02:57:35.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:35.408+0000 7f0cd10aa640 1 -- 172.21.15.16:0/3686247648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ccc066410 msgr2=0x7f0ccc0fc8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:57:35.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:35.408+0000 7f0cd10aa640 1 --2- 172.21.15.16:0/3686247648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ccc066410 0x7f0ccc0fc8c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc000c00 tx=0x7f0cbc009da0 comp rx=0 tx=0).stop 2024-06-18T02:57:35.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:35.409+0000 7f0cd10aa640 1 -- 172.21.15.16:0/3686247648 shutdown_connections 2024-06-18T02:57:35.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:35.409+0000 7f0cd10aa640 1 --2- 172.21.15.16:0/3686247648 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f0cb4065430 0x7f0cb40678e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:57:35.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:35.409+0000 7f0cd10aa640 1 --2- 172.21.15.16:0/3686247648 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ccc0ff3f0 0x7f0ccc0fadb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:57:35.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:35.409+0000 7f0cd10aa640 1 --2- 172.21.15.16:0/3686247648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ccc066410 0x7f0ccc0fc8c0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:57:35.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:35.409+0000 7f0cd10aa640 1 -- 172.21.15.16:0/3686247648 >> 172.21.15.16:0/3686247648 conn(0x7f0ccc0f6980 msgr2=0x7f0ccc0f6d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:57:35.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:35.409+0000 7f0cd10aa640 1 -- 172.21.15.16:0/3686247648 shutdown_connections 2024-06-18T02:57:35.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:35.409+0000 7f0cd10aa640 1 -- 172.21.15.16:0/3686247648 wait complete. 2024-06-18T02:57:35.536 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:35 smithi016 ceph-mon[29214]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-06-18T02:57:35.537 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:35 smithi016 ceph-mon[29214]: osdmap e39: 6 total, 5 up, 6 in 2024-06-18T02:57:35.537 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:35 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-18T02:57:35.537 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:35 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-18T02:57:35.537 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:35 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/3686247648' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-06-18T02:57:35.727 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:35 smithi123 ceph-mon[36358]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi123", "root=default"]}]': finished 2024-06-18T02:57:35.727 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:35 smithi123 ceph-mon[36358]: osdmap e39: 6 total, 5 up, 6 in 2024-06-18T02:57:35.727 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:35 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-18T02:57:35.727 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:35 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-18T02:57:35.727 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:35 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/3686247648' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-06-18T02:57:36.044 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":39,"num_osds":6,"num_up_osds":5,"osd_up_since":1718679430,"num_in_osds":6,"osd_in_since":1718679436,"num_remapped_pgs":0} 2024-06-18T02:57:36.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:36 smithi123 ceph-mon[36358]: purged_snaps scrub starts 2024-06-18T02:57:36.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:36 smithi123 ceph-mon[36358]: purged_snaps scrub ok 2024-06-18T02:57:36.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:36 smithi123 ceph-mon[36358]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-18T02:57:36.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:36 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-18T02:57:36.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:36 smithi123 ceph-mon[36358]: osd.5 [v2:172.21.15.123:6816/4116323333,v1:172.21.15.123:6817/4116323333] boot 2024-06-18T02:57:36.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:36 smithi123 ceph-mon[36358]: osdmap e40: 6 total, 6 up, 6 in 2024-06-18T02:57:36.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:36 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-18T02:57:36.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:36 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:57:36.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:36 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:57:36.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:36 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-18T02:57:36.736 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:36 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-18T02:57:36.736 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:36 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-18T02:57:36.736 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:36 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:57:36.736 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:36 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:57:36.736 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:36 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T02:57:36.736 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:36 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:57:36.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:36 smithi016 ceph-mon[29214]: purged_snaps scrub starts 2024-06-18T02:57:36.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:36 smithi016 ceph-mon[29214]: purged_snaps scrub ok 2024-06-18T02:57:36.806 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:36 smithi016 ceph-mon[29214]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-18T02:57:36.806 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:36 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-18T02:57:36.806 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:36 smithi016 ceph-mon[29214]: osd.5 [v2:172.21.15.123:6816/4116323333,v1:172.21.15.123:6817/4116323333] boot 2024-06-18T02:57:36.806 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:36 smithi016 ceph-mon[29214]: osdmap e40: 6 total, 6 up, 6 in 2024-06-18T02:57:36.806 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:36 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-18T02:57:36.806 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:36 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:57:36.806 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:36 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:57:36.806 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:36 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-18T02:57:36.806 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:36 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-18T02:57:36.806 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:36 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-18T02:57:36.806 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:36 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:57:36.806 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:36 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:57:36.807 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:36 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T02:57:36.807 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:36 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:57:37.045 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph osd stat -f json 2024-06-18T02:57:37.315 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T02:57:37.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:37 smithi123 ceph-mon[36358]: Detected new or changed devices on smithi123 2024-06-18T02:57:37.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:37 smithi123 ceph-mon[36358]: Adjusting osd_memory_target on smithi123 to 5312M 2024-06-18T02:57:37.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:37 smithi123 ceph-mon[36358]: osdmap e41: 6 total, 6 up, 6 in 2024-06-18T02:57:37.779 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:37 smithi016 ceph-mon[29214]: Detected new or changed devices on smithi123 2024-06-18T02:57:37.780 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:37 smithi016 ceph-mon[29214]: Adjusting osd_memory_target on smithi123 to 5312M 2024-06-18T02:57:37.780 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:37 smithi016 ceph-mon[29214]: osdmap e41: 6 total, 6 up, 6 in 2024-06-18T02:57:38.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:38 smithi123 ceph-mon[36358]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:57:38.779 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:38 smithi016 ceph-mon[29214]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:57:38.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:38.985+0000 7f4a58bd5640 1 -- 172.21.15.16:0/3167810517 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a540fd920 msgr2=0x7f4a540fdd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:57:38.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:38.985+0000 7f4a58bd5640 1 --2- 172.21.15.16:0/3167810517 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a540fd920 0x7f4a540fdd30 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f4a440048d0 tx=0x7f4a44009da0 comp rx=0 tx=0).stop 2024-06-18T02:57:38.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:38.985+0000 7f4a58bd5640 1 -- 172.21.15.16:0/3167810517 shutdown_connections 2024-06-18T02:57:38.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:38.985+0000 7f4a58bd5640 1 --2- 172.21.15.16:0/3167810517 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4a540fe270 0x7f4a540fa3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:57:38.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:38.985+0000 7f4a58bd5640 1 --2- 172.21.15.16:0/3167810517 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a540fd920 0x7f4a540fdd30 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:57:38.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:38.985+0000 7f4a58bd5640 1 -- 172.21.15.16:0/3167810517 >> 172.21.15.16:0/3167810517 conn(0x7f4a540f6120 msgr2=0x7f4a540f8550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:57:38.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:38.985+0000 7f4a58bd5640 1 -- 172.21.15.16:0/3167810517 shutdown_connections 2024-06-18T02:57:38.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:38.985+0000 7f4a58bd5640 1 -- 172.21.15.16:0/3167810517 wait complete. 2024-06-18T02:57:38.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:38.986+0000 7f4a58bd5640 1 Processor -- start 2024-06-18T02:57:38.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:38.986+0000 7f4a58bd5640 1 -- start start 2024-06-18T02:57:38.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:38.986+0000 7f4a58bd5640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4a540fd920 0x7f4a540fd0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:57:38.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:38.986+0000 7f4a58bd5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a540fe270 0x7f4a540fb600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:57:38.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:38.986+0000 7f4a58bd5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a540fbb40 con 0x7f4a540fe270 2024-06-18T02:57:38.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:38.986+0000 7f4a58bd5640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a540fbcb0 con 0x7f4a540fd920 2024-06-18T02:57:38.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:38.986+0000 7f4a49d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a540fe270 0x7f4a540fb600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:57:38.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:38.987+0000 7f4a49d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a540fe270 0x7f4a540fb600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48254/0 (socket says 172.21.15.16:48254) 2024-06-18T02:57:38.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:38.987+0000 7f4a49d9b640 1 -- 172.21.15.16:0/704757962 learned_addr learned my addr 172.21.15.16:0/704757962 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:57:38.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:38.987+0000 7f4a5259c640 1 --2- 172.21.15.16:0/704757962 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4a540fd920 0x7f4a540fd0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:57:38.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:38.987+0000 7f4a49d9b640 1 -- 172.21.15.16:0/704757962 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4a540fd920 msgr2=0x7f4a540fd0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:57:38.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:38.987+0000 7f4a49d9b640 1 --2- 172.21.15.16:0/704757962 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4a540fd920 0x7f4a540fd0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:57:38.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:38.987+0000 7f4a49d9b640 1 -- 172.21.15.16:0/704757962 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a44005040 con 0x7f4a540fe270 2024-06-18T02:57:38.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:38.987+0000 7f4a49d9b640 1 --2- 172.21.15.16:0/704757962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a540fe270 0x7f4a540fb600 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4a3c002f40 tx=0x7f4a3c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:57:38.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:38.987+0000 7f4a4b7fe640 1 -- 172.21.15.16:0/704757962 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4a3c007c30 con 0x7f4a540fe270 2024-06-18T02:57:38.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:38.987+0000 7f4a58bd5640 1 -- 172.21.15.16:0/704757962 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a540fbee0 con 0x7f4a540fe270 2024-06-18T02:57:38.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:38.987+0000 7f4a4b7fe640 1 -- 172.21.15.16:0/704757962 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4a3c00f040 con 0x7f4a540fe270 2024-06-18T02:57:38.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:38.987+0000 7f4a4b7fe640 1 -- 172.21.15.16:0/704757962 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4a3c00cc30 con 0x7f4a540fe270 2024-06-18T02:57:38.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:38.987+0000 7f4a58bd5640 1 -- 172.21.15.16:0/704757962 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a540fc450 con 0x7f4a540fe270 2024-06-18T02:57:38.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:38.988+0000 7f4a58bd5640 1 -- 172.21.15.16:0/704757962 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a540f9dc0 con 0x7f4a540fe270 2024-06-18T02:57:38.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:38.989+0000 7f4a4b7fe640 1 -- 172.21.15.16:0/704757962 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f4a3c0090d0 con 0x7f4a540fe270 2024-06-18T02:57:38.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:38.989+0000 7f4a4b7fe640 1 --2- 172.21.15.16:0/704757962 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f4a40065430 0x7f4a400678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:57:38.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:38.989+0000 7f4a4b7fe640 1 -- 172.21.15.16:0/704757962 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f4a3c0820d0 con 0x7f4a540fe270 2024-06-18T02:57:38.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:38.989+0000 7f4a5259c640 1 --2- 172.21.15.16:0/704757962 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f4a40065430 0x7f4a400678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:57:38.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:38.990+0000 7f4a5259c640 1 --2- 172.21.15.16:0/704757962 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f4a40065430 0x7f4a400678e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4a44009f80 tx=0x7f4a44035040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:57:38.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:38.994+0000 7f4a4b7fe640 1 -- 172.21.15.16:0/704757962 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f4a3c050630 con 0x7f4a540fe270 2024-06-18T02:57:39.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:39.166+0000 7f4a58bd5640 1 -- 172.21.15.16:0/704757962 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f4a5404b480 con 0x7f4a540fe270 2024-06-18T02:57:39.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:39.166+0000 7f4a4b7fe640 1 -- 172.21.15.16:0/704757962 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v41) v1 ==== 74+0+130 (secure 0 0 0) 0x7f4a3c053c50 con 0x7f4a540fe270 2024-06-18T02:57:39.168 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:57:39.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:39.168+0000 7f4a58bd5640 1 -- 172.21.15.16:0/704757962 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f4a40065430 msgr2=0x7f4a400678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:57:39.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:39.168+0000 7f4a58bd5640 1 --2- 172.21.15.16:0/704757962 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f4a40065430 0x7f4a400678e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4a44009f80 tx=0x7f4a44035040 comp rx=0 tx=0).stop 2024-06-18T02:57:39.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:39.168+0000 7f4a58bd5640 1 -- 172.21.15.16:0/704757962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a540fe270 msgr2=0x7f4a540fb600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:57:39.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:39.168+0000 7f4a58bd5640 1 --2- 172.21.15.16:0/704757962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a540fe270 0x7f4a540fb600 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4a3c002f40 tx=0x7f4a3c00b040 comp rx=0 tx=0).stop 2024-06-18T02:57:39.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:39.168+0000 7f4a58bd5640 1 -- 172.21.15.16:0/704757962 shutdown_connections 2024-06-18T02:57:39.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:39.168+0000 7f4a58bd5640 1 --2- 172.21.15.16:0/704757962 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f4a40065430 0x7f4a400678e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:57:39.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:39.168+0000 7f4a58bd5640 1 --2- 172.21.15.16:0/704757962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a540fe270 0x7f4a540fb600 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:57:39.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:39.168+0000 7f4a58bd5640 1 --2- 172.21.15.16:0/704757962 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4a540fd920 0x7f4a540fd0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:57:39.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:39.168+0000 7f4a58bd5640 1 -- 172.21.15.16:0/704757962 >> 172.21.15.16:0/704757962 conn(0x7f4a540f6120 msgr2=0x7f4a540f7b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:57:39.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:39.168+0000 7f4a58bd5640 1 -- 172.21.15.16:0/704757962 shutdown_connections 2024-06-18T02:57:39.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:39.168+0000 7f4a58bd5640 1 -- 172.21.15.16:0/704757962 wait complete. 2024-06-18T02:57:39.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:39 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/704757962' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-06-18T02:57:39.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:39 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/704757962' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-06-18T02:57:39.911 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":41,"num_osds":6,"num_up_osds":6,"osd_up_since":1718679455,"num_in_osds":6,"osd_in_since":1718679436,"num_remapped_pgs":0} 2024-06-18T02:57:39.912 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph osd dump --format=json 2024-06-18T02:57:40.180 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T02:57:40.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:40 smithi016 ceph-mon[29214]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:57:40.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:40 smithi123 ceph-mon[36358]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:57:41.604 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:41 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:57:41.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:41 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:57:41.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:41.810+0000 7f86b65eb640 1 -- 172.21.15.16:0/1668782496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86b01022c0 msgr2=0x7f86b0102690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:57:41.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:41.810+0000 7f86b65eb640 1 --2- 172.21.15.16:0/1668782496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86b01022c0 0x7f86b0102690 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f86a00048d0 tx=0x7f86a0009da0 comp rx=0 tx=0).stop 2024-06-18T02:57:41.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:41.810+0000 7f86b65eb640 1 -- 172.21.15.16:0/1668782496 shutdown_connections 2024-06-18T02:57:41.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:41.810+0000 7f86b65eb640 1 --2- 172.21.15.16:0/1668782496 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f86b0102c60 0x7f86b00fc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:57:41.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:41.810+0000 7f86b65eb640 1 --2- 172.21.15.16:0/1668782496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86b01022c0 0x7f86b0102690 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:57:41.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:41.810+0000 7f86b65eb640 1 -- 172.21.15.16:0/1668782496 >> 172.21.15.16:0/1668782496 conn(0x7f86b00f7e50 msgr2=0x7f86b00fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:57:41.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:41.810+0000 7f86b65eb640 1 -- 172.21.15.16:0/1668782496 shutdown_connections 2024-06-18T02:57:41.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:41.810+0000 7f86b65eb640 1 -- 172.21.15.16:0/1668782496 wait complete. 2024-06-18T02:57:41.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:41.810+0000 7f86b65eb640 1 Processor -- start 2024-06-18T02:57:41.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:41.811+0000 7f86b65eb640 1 -- start start 2024-06-18T02:57:41.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:41.811+0000 7f86b65eb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86b01022c0 0x7f86b0113330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:57:41.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:41.811+0000 7f86b65eb640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f86b0102c60 0x7f86b0113870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:57:41.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:41.811+0000 7f86b65eb640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86b0113e70 con 0x7f86b01022c0 2024-06-18T02:57:41.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:41.811+0000 7f86b65eb640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86b010d2b0 con 0x7f86b0102c60 2024-06-18T02:57:41.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:41.811+0000 7f86affff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86b01022c0 0x7f86b0113330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:57:41.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:41.811+0000 7f86affff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86b01022c0 0x7f86b0113330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41286/0 (socket says 172.21.15.16:41286) 2024-06-18T02:57:41.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:41.811+0000 7f86affff640 1 -- 172.21.15.16:0/3786150790 learned_addr learned my addr 172.21.15.16:0/3786150790 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:57:41.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:41.811+0000 7f86af7fe640 1 --2- 172.21.15.16:0/3786150790 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f86b0102c60 0x7f86b0113870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:57:41.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:41.811+0000 7f86affff640 1 -- 172.21.15.16:0/3786150790 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f86b0102c60 msgr2=0x7f86b0113870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:57:41.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:41.811+0000 7f86affff640 1 --2- 172.21.15.16:0/3786150790 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f86b0102c60 0x7f86b0113870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:57:41.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:41.811+0000 7f86affff640 1 -- 172.21.15.16:0/3786150790 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86a0005040 con 0x7f86b01022c0 2024-06-18T02:57:41.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:41.811+0000 7f86af7fe640 1 --2- 172.21.15.16:0/3786150790 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f86b0102c60 0x7f86b0113870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-18T02:57:41.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:41.811+0000 7f86affff640 1 --2- 172.21.15.16:0/3786150790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86b01022c0 0x7f86b0113330 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f86a0003080 tx=0x7f86a0009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:57:41.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:41.812+0000 7f86acff9640 1 -- 172.21.15.16:0/3786150790 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f86a0038070 con 0x7f86b01022c0 2024-06-18T02:57:41.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:41.812+0000 7f86b65eb640 1 -- 172.21.15.16:0/3786150790 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86b010d530 con 0x7f86b01022c0 2024-06-18T02:57:41.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:41.812+0000 7f86acff9640 1 -- 172.21.15.16:0/3786150790 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f86a0032970 con 0x7f86b01022c0 2024-06-18T02:57:41.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:41.812+0000 7f86acff9640 1 -- 172.21.15.16:0/3786150790 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f86a003d960 con 0x7f86b01022c0 2024-06-18T02:57:41.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:41.812+0000 7f86b65eb640 1 -- 172.21.15.16:0/3786150790 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86b010da70 con 0x7f86b01022c0 2024-06-18T02:57:41.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:41.813+0000 7f86b65eb640 1 -- 172.21.15.16:0/3786150790 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86b00fda30 con 0x7f86b01022c0 2024-06-18T02:57:41.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:41.813+0000 7f86acff9640 1 -- 172.21.15.16:0/3786150790 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f86a0030020 con 0x7f86b01022c0 2024-06-18T02:57:41.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:41.814+0000 7f86acff9640 1 --2- 172.21.15.16:0/3786150790 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f8698065430 0x7f86980678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:57:41.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:41.814+0000 7f86acff9640 1 -- 172.21.15.16:0/3786150790 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f86a00a26d0 con 0x7f86b01022c0 2024-06-18T02:57:41.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:41.814+0000 7f86af7fe640 1 --2- 172.21.15.16:0/3786150790 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f8698065430 0x7f86980678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:57:41.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:41.814+0000 7f86af7fe640 1 --2- 172.21.15.16:0/3786150790 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f8698065430 0x7f86980678e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f86b010ebf0 tx=0x7f86a400b040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:57:41.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:41.818+0000 7f86acff9640 1 -- 172.21.15.16:0/3786150790 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f86a0070ae0 con 0x7f86b01022c0 2024-06-18T02:57:41.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:41.996+0000 7f86b65eb640 1 -- 172.21.15.16:0/3786150790 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f86b00fbab0 con 0x7f86b01022c0 2024-06-18T02:57:41.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:41.996+0000 7f86acff9640 1 -- 172.21.15.16:0/3786150790 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+11083 (secure 0 0 0) 0x7f86a0074100 con 0x7f86b01022c0 2024-06-18T02:57:41.997 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:57:41.997 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":41,"fsid":"9e93709a-2d1d-11ef-bc9d-c7b262605968","created":"2024-06-18T02:51:57.320739+0000","modified":"2024-06-18T02:57:36.450020+0000","last_up_change":"2024-06-18T02:57:35.446077+0000","last_in_change":"2024-06-18T02:57:16.677249+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-18T02:56:23.704246+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"75595aa3-a41d-4081-9e6a-e564de1b58ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":3634864038},{"type":"v1","addr":"172.21.15.16:6803","nonce":3634864038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":3634864038},{"type":"v1","addr":"172.21.15.16:6805","nonce":3634864038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":3634864038},{"type":"v1","addr":"172.21.15.16:6809","nonce":3634864038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":3634864038},{"type":"v1","addr":"172.21.15.16:6807","nonce":3634864038}]},"public_addr":"172.21.15.16:6803/3634864038","cluster_addr":"172.21.15.16:6805/3634864038","heartbeat_back_addr":"172.21.15.16:6809/3634864038","heartbeat_front_addr":"172.21.15.16:6807/3634864038","state":["exists","up"]},{"osd":1,"uuid":"33accf47-d8b6-4b00-b579-2897f47e381d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":500143512},{"type":"v1","addr":"172.21.15.16:6811","nonce":500143512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":500143512},{"type":"v1","addr":"172.21.15.16:6813","nonce":500143512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":500143512},{"type":"v1","addr":"172.21.15.16:6817","nonce":500143512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":500143512},{"type":"v1","addr":"172.21.15.16:6815","nonce":500143512}]},"public_addr":"172.21.15.16:6811/500143512","cluster_addr":"172.21.15.16:6813/500143512","heartbeat_back_addr":"172.21.15.16:6817/500143512","heartbeat_front_addr":"172.21.15.16:6815/500143512","state":["exists","up"]},{"osd":2,"uuid":"f9d60224-3bc7-458c-a8b7-8384a7e219d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1319853338},{"type":"v1","addr":"172.21.15.16:6819","nonce":1319853338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":1319853338},{"type":"v1","addr":"172.21.15.16:6821","nonce":1319853338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":1319853338},{"type":"v1","addr":"172.21.15.16:6825","nonce":1319853338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":1319853338},{"type":"v1","addr":"172.21.15.16:6823","nonce":1319853338}]},"public_addr":"172.21.15.16:6819/1319853338","cluster_addr":"172.21.15.16:6821/1319853338","heartbeat_back_addr":"172.21.15.16:6825/1319853338","heartbeat_front_addr":"172.21.15.16:6823/1319853338","state":["exists","up"]},{"osd":3,"uuid":"d432cb55-356d-44f1-b6f0-cc98fbec6809","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":2187102998},{"type":"v1","addr":"172.21.15.123:6801","nonce":2187102998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":2187102998},{"type":"v1","addr":"172.21.15.123:6803","nonce":2187102998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":2187102998},{"type":"v1","addr":"172.21.15.123:6807","nonce":2187102998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":2187102998},{"type":"v1","addr":"172.21.15.123:6805","nonce":2187102998}]},"public_addr":"172.21.15.123:6801/2187102998","cluster_addr":"172.21.15.123:6803/2187102998","heartbeat_back_addr":"172.21.15.123:6807/2187102998","heartbeat_front_addr":"172.21.15.123:6805/2187102998","state":["exists","up"]},{"osd":4,"uuid":"05cf06af-331b-408e-a7b4-1ae1486c74a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":4038205871},{"type":"v1","addr":"172.21.15.123:6809","nonce":4038205871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":4038205871},{"type":"v1","addr":"172.21.15.123:6811","nonce":4038205871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":4038205871},{"type":"v1","addr":"172.21.15.123:6815","nonce":4038205871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":4038205871},{"type":"v1","addr":"172.21.15.123:6813","nonce":4038205871}]},"public_addr":"172.21.15.123:6809/4038205871","cluster_addr":"172.21.15.123:6811/4038205871","heartbeat_back_addr":"172.21.15.123:6815/4038205871","heartbeat_front_addr":"172.21.15.123:6813/4038205871","state":["exists","up"]},{"osd":5,"uuid":"faa63f45-ee41-485f-afad-9762cbee8f82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4116323333},{"type":"v1","addr":"172.21.15.123:6817","nonce":4116323333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4116323333},{"type":"v1","addr":"172.21.15.123:6819","nonce":4116323333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4116323333},{"type":"v1","addr":"172.21.15.123:6823","nonce":4116323333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4116323333},{"type":"v1","addr":"172.21.15.123:6821","nonce":4116323333}]},"public_addr":"172.21.15.123:6817/4116323333","cluster_addr":"172.21.15.123:6819/4116323333","heartbeat_back_addr":"172.21.15.123:6823/4116323333","heartbeat_front_addr":"172.21.15.123:6821/4116323333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-18T02:55:25.559702+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-18T02:55:51.840436+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-18T02:56:19.314038+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-18T02:56:42.662054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-18T02:57:09.162120+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-18T02:57:33.526602+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/684280009":"2024-06-19T02:54:11.194734+0000","172.21.15.16:0/1097020524":"2024-06-19T02:52:27.324621+0000","172.21.15.16:6800/1678074233":"2024-06-19T02:52:27.324621+0000","172.21.15.16:0/1691231433":"2024-06-19T02:52:27.324621+0000","172.21.15.16:6801/1678074233":"2024-06-19T02:52:27.324621+0000","172.21.15.16:0/904626067":"2024-06-19T02:52:58.168970+0000","172.21.15.16:6800/1807632468":"2024-06-19T02:52:58.168970+0000","172.21.15.16:0/4088735442":"2024-06-19T02:54:11.194734+0000","172.21.15.16:0/1001205060":"2024-06-19T02:52:27.324621+0000","172.21.15.16:0/3691774959":"2024-06-19T02:52:58.168970+0000","172.21.15.16:6801/1807632468":"2024-06-19T02:52:58.168970+0000","172.21.15.16:0/3097100123":"2024-06-19T02:52:58.168970+0000","172.21.15.16:0/2121279768":"2024-06-19T02:54:11.194734+0000","172.21.15.16:6800/3749129160":"2024-06-19T02:54:11.194734+0000","172.21.15.16:6801/3749129160":"2024-06-19T02:54:11.194734+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-18T02:57:41.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:41.998+0000 7f86b65eb640 1 -- 172.21.15.16:0/3786150790 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f8698065430 msgr2=0x7f86980678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:57:41.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:41.998+0000 7f86b65eb640 1 --2- 172.21.15.16:0/3786150790 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f8698065430 0x7f86980678e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f86b010ebf0 tx=0x7f86a400b040 comp rx=0 tx=0).stop 2024-06-18T02:57:41.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:41.998+0000 7f86b65eb640 1 -- 172.21.15.16:0/3786150790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86b01022c0 msgr2=0x7f86b0113330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:57:41.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:41.998+0000 7f86b65eb640 1 --2- 172.21.15.16:0/3786150790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86b01022c0 0x7f86b0113330 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f86a0003080 tx=0x7f86a0009da0 comp rx=0 tx=0).stop 2024-06-18T02:57:41.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:41.999+0000 7f86b65eb640 1 -- 172.21.15.16:0/3786150790 shutdown_connections 2024-06-18T02:57:41.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:41.999+0000 7f86b65eb640 1 --2- 172.21.15.16:0/3786150790 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f8698065430 0x7f86980678e0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:57:41.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:41.999+0000 7f86b65eb640 1 --2- 172.21.15.16:0/3786150790 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f86b0102c60 0x7f86b0113870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:57:42.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:41.999+0000 7f86b65eb640 1 --2- 172.21.15.16:0/3786150790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f86b01022c0 0x7f86b0113330 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:57:42.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:41.999+0000 7f86b65eb640 1 -- 172.21.15.16:0/3786150790 >> 172.21.15.16:0/3786150790 conn(0x7f86b00f7e50 msgr2=0x7f86b00f8dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:57:42.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:41.999+0000 7f86b65eb640 1 -- 172.21.15.16:0/3786150790 shutdown_connections 2024-06-18T02:57:42.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:41.999+0000 7f86b65eb640 1 -- 172.21.15.16:0/3786150790 wait complete. 2024-06-18T02:57:42.664 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:42 smithi016 ceph-mon[29214]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:57:42.665 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:42 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/3786150790' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-18T02:57:42.696 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-18T02:56:23.704246+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-06-18T02:57:42.696 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-06-18T02:57:42.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:42 smithi123 ceph-mon[36358]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:57:42.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:42 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/3786150790' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-18T02:57:42.966 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T02:57:44.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:44.650+0000 7f7b9ad07640 1 -- 172.21.15.16:0/202512198 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b940fa9e0 msgr2=0x7f7b940fae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:57:44.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:44.650+0000 7f7b9ad07640 1 --2- 172.21.15.16:0/202512198 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b940fa9e0 0x7f7b940fae50 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f7b84009c50 tx=0x7f7b84004d10 comp rx=0 tx=0).stop 2024-06-18T02:57:44.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:44.650+0000 7f7b9ad07640 1 -- 172.21.15.16:0/202512198 shutdown_connections 2024-06-18T02:57:44.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:44.650+0000 7f7b9ad07640 1 --2- 172.21.15.16:0/202512198 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b940fa9e0 0x7f7b940fae50 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:57:44.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:44.650+0000 7f7b9ad07640 1 --2- 172.21.15.16:0/202512198 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b94102370 0x7f7b94102740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:57:44.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:44.650+0000 7f7b9ad07640 1 -- 172.21.15.16:0/202512198 >> 172.21.15.16:0/202512198 conn(0x7f7b94070a40 msgr2=0x7f7b94070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:57:44.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:44.650+0000 7f7b9ad07640 1 -- 172.21.15.16:0/202512198 shutdown_connections 2024-06-18T02:57:44.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:44.650+0000 7f7b9ad07640 1 -- 172.21.15.16:0/202512198 wait complete. 2024-06-18T02:57:44.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:44.651+0000 7f7b9ad07640 1 Processor -- start 2024-06-18T02:57:44.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:44.651+0000 7f7b9ad07640 1 -- start start 2024-06-18T02:57:44.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:44.651+0000 7f7b9ad07640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b940fa9e0 0x7f7b940fbbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:57:44.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:44.651+0000 7f7b9ad07640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b94102370 0x7f7b940fc100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:57:44.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:44.651+0000 7f7b9ad07640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b940fc640 con 0x7f7b940fa9e0 2024-06-18T02:57:44.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:44.651+0000 7f7b9ad07640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b940fc7b0 con 0x7f7b94102370 2024-06-18T02:57:44.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:44.651+0000 7f7b98aa3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b940fa9e0 0x7f7b940fbbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:57:44.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:44.651+0000 7f7b98aa3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b940fa9e0 0x7f7b940fbbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41302/0 (socket says 172.21.15.16:41302) 2024-06-18T02:57:44.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:44.651+0000 7f7b98aa3640 1 -- 172.21.15.16:0/3903623766 learned_addr learned my addr 172.21.15.16:0/3903623766 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:57:44.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:44.651+0000 7f7b93fff640 1 --2- 172.21.15.16:0/3903623766 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b94102370 0x7f7b940fc100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:57:44.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:44.652+0000 7f7b98aa3640 1 -- 172.21.15.16:0/3903623766 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b94102370 msgr2=0x7f7b940fc100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:57:44.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:44.652+0000 7f7b98aa3640 1 --2- 172.21.15.16:0/3903623766 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b94102370 0x7f7b940fc100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:57:44.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:44.652+0000 7f7b98aa3640 1 -- 172.21.15.16:0/3903623766 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b840098b0 con 0x7f7b940fa9e0 2024-06-18T02:57:44.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:44.652+0000 7f7b93fff640 1 --2- 172.21.15.16:0/3903623766 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b94102370 0x7f7b940fc100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-18T02:57:44.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:44.652+0000 7f7b98aa3640 1 --2- 172.21.15.16:0/3903623766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b940fa9e0 0x7f7b940fbbc0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c002a50 tx=0x7f7b7c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:57:44.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:44.652+0000 7f7b917fa640 1 -- 172.21.15.16:0/3903623766 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b7c007c30 con 0x7f7b940fa9e0 2024-06-18T02:57:44.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:44.652+0000 7f7b9ad07640 1 -- 172.21.15.16:0/3903623766 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b940fca40 con 0x7f7b940fa9e0 2024-06-18T02:57:44.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:44.652+0000 7f7b917fa640 1 -- 172.21.15.16:0/3903623766 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7b7c00f040 con 0x7f7b940fa9e0 2024-06-18T02:57:44.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:44.652+0000 7f7b917fa640 1 -- 172.21.15.16:0/3903623766 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b7c00db50 con 0x7f7b940fa9e0 2024-06-18T02:57:44.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:44.652+0000 7f7b9ad07640 1 -- 172.21.15.16:0/3903623766 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b941196a0 con 0x7f7b940fa9e0 2024-06-18T02:57:44.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:44.653+0000 7f7b9ad07640 1 -- 172.21.15.16:0/3903623766 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b94065530 con 0x7f7b940fa9e0 2024-06-18T02:57:44.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:44.654+0000 7f7b917fa640 1 -- 172.21.15.16:0/3903623766 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f7b7c01c400 con 0x7f7b940fa9e0 2024-06-18T02:57:44.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:44.654+0000 7f7b917fa640 1 --2- 172.21.15.16:0/3903623766 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f7b80065430 0x7f7b800678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:57:44.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:44.654+0000 7f7b917fa640 1 -- 172.21.15.16:0/3903623766 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f7b7c081e00 con 0x7f7b940fa9e0 2024-06-18T02:57:44.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:44.654+0000 7f7b93fff640 1 --2- 172.21.15.16:0/3903623766 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f7b80065430 0x7f7b800678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:57:44.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:44.655+0000 7f7b93fff640 1 --2- 172.21.15.16:0/3903623766 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f7b80065430 0x7f7b800678e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7b84000f80 tx=0x7f7b84004d10 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:57:44.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:44.658+0000 7f7b917fa640 1 -- 172.21.15.16:0/3903623766 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f7b7c0502b0 con 0x7f7b940fa9e0 2024-06-18T02:57:44.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:44 smithi123 ceph-mon[36358]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:57:44.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:44 smithi016 ceph-mon[29214]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:57:44.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:44.833+0000 7f7b9ad07640 1 -- 172.21.15.16:0/3903623766 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7b940fd2b0 con 0x7f7b940fa9e0 2024-06-18T02:57:44.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:44.833+0000 7f7b917fa640 1 -- 172.21.15.16:0/3903623766 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v41) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7b7c0538d0 con 0x7f7b940fa9e0 2024-06-18T02:57:44.834 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-06-18T02:57:44.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:44.835+0000 7f7b9ad07640 1 -- 172.21.15.16:0/3903623766 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f7b80065430 msgr2=0x7f7b800678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:57:44.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:44.835+0000 7f7b9ad07640 1 --2- 172.21.15.16:0/3903623766 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f7b80065430 0x7f7b800678e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7b84000f80 tx=0x7f7b84004d10 comp rx=0 tx=0).stop 2024-06-18T02:57:44.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:44.835+0000 7f7b9ad07640 1 -- 172.21.15.16:0/3903623766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b940fa9e0 msgr2=0x7f7b940fbbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:57:44.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:44.835+0000 7f7b9ad07640 1 --2- 172.21.15.16:0/3903623766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b940fa9e0 0x7f7b940fbbc0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c002a50 tx=0x7f7b7c004600 comp rx=0 tx=0).stop 2024-06-18T02:57:44.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:44.835+0000 7f7b9ad07640 1 -- 172.21.15.16:0/3903623766 shutdown_connections 2024-06-18T02:57:44.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:44.835+0000 7f7b9ad07640 1 --2- 172.21.15.16:0/3903623766 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f7b80065430 0x7f7b800678e0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:57:44.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:44.835+0000 7f7b9ad07640 1 --2- 172.21.15.16:0/3903623766 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7b94102370 0x7f7b940fc100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:57:44.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:44.835+0000 7f7b9ad07640 1 --2- 172.21.15.16:0/3903623766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b940fa9e0 0x7f7b940fbbc0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:57:44.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:44.835+0000 7f7b9ad07640 1 -- 172.21.15.16:0/3903623766 >> 172.21.15.16:0/3903623766 conn(0x7f7b94070a40 msgr2=0x7f7b940f9900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:57:44.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:44.835+0000 7f7b9ad07640 1 -- 172.21.15.16:0/3903623766 shutdown_connections 2024-06-18T02:57:44.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:44.835+0000 7f7b9ad07640 1 -- 172.21.15.16:0/3903623766 wait complete. 2024-06-18T02:57:45.567 INFO:tasks.cephadm:Setting up client nodes... 2024-06-18T02:57:45.568 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-06-18T02:57:45.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:45 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/3903623766' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-06-18T02:57:45.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:45 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/3903623766' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-06-18T02:57:45.837 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T02:57:46.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:46 smithi123 ceph-mon[36358]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:57:46.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:46 smithi016 ceph-mon[29214]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:57:47.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:47.508+0000 7f4734309640 1 -- 172.21.15.16:0/3472720865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f472c0ff420 msgr2=0x7f472c101800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:57:47.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:47.508+0000 7f4734309640 1 --2- 172.21.15.16:0/3472720865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f472c0ff420 0x7f472c101800 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f47200048d0 tx=0x7f4720009da0 comp rx=0 tx=0).stop 2024-06-18T02:57:47.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:47.508+0000 7f4734309640 1 -- 172.21.15.16:0/3472720865 shutdown_connections 2024-06-18T02:57:47.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:47.508+0000 7f4734309640 1 --2- 172.21.15.16:0/3472720865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f472c0ff420 0x7f472c101800 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:57:47.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:47.508+0000 7f4734309640 1 --2- 172.21.15.16:0/3472720865 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f472c066410 0x7f472c0feee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:57:47.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:47.508+0000 7f4734309640 1 -- 172.21.15.16:0/3472720865 >> 172.21.15.16:0/3472720865 conn(0x7f472c0f6990 msgr2=0x7f472c0f8dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:57:47.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:47.508+0000 7f4734309640 1 -- 172.21.15.16:0/3472720865 shutdown_connections 2024-06-18T02:57:47.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:47.508+0000 7f4734309640 1 -- 172.21.15.16:0/3472720865 wait complete. 2024-06-18T02:57:47.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:47.509+0000 7f4734309640 1 Processor -- start 2024-06-18T02:57:47.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:47.509+0000 7f4734309640 1 -- start start 2024-06-18T02:57:47.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:47.509+0000 7f4734309640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f472c066410 0x7f472c0fc8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:57:47.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:47.509+0000 7f4734309640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f472c0ff420 0x7f472c0fade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:57:47.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:47.509+0000 7f4734309640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f472c0fb370 con 0x7f472c0ff420 2024-06-18T02:57:47.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:47.509+0000 7f4734309640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f472c0fb4b0 con 0x7f472c066410 2024-06-18T02:57:47.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:47.509+0000 7f47318a4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f472c0ff420 0x7f472c0fade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:57:47.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:47.509+0000 7f47318a4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f472c0ff420 0x7f472c0fade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:41330/0 (socket says 172.21.15.16:41330) 2024-06-18T02:57:47.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:47.509+0000 7f47318a4640 1 -- 172.21.15.16:0/2013494329 learned_addr learned my addr 172.21.15.16:0/2013494329 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:57:47.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:47.509+0000 7f47320a5640 1 --2- 172.21.15.16:0/2013494329 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f472c066410 0x7f472c0fc8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:57:47.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:47.510+0000 7f47318a4640 1 -- 172.21.15.16:0/2013494329 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f472c066410 msgr2=0x7f472c0fc8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:57:47.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:47.510+0000 7f47318a4640 1 --2- 172.21.15.16:0/2013494329 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f472c066410 0x7f472c0fc8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:57:47.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:47.510+0000 7f47318a4640 1 -- 172.21.15.16:0/2013494329 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4720005040 con 0x7f472c0ff420 2024-06-18T02:57:47.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:47.510+0000 7f47318a4640 1 --2- 172.21.15.16:0/2013494329 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f472c0ff420 0x7f472c0fade0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f4720009f80 tx=0x7f4720009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:57:47.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:47.510+0000 7f471effd640 1 -- 172.21.15.16:0/2013494329 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4720038070 con 0x7f472c0ff420 2024-06-18T02:57:47.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:47.510+0000 7f471effd640 1 -- 172.21.15.16:0/2013494329 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4720008ee0 con 0x7f472c0ff420 2024-06-18T02:57:47.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:47.510+0000 7f4734309640 1 -- 172.21.15.16:0/2013494329 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f472c0fb730 con 0x7f472c0ff420 2024-06-18T02:57:47.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:47.510+0000 7f471effd640 1 -- 172.21.15.16:0/2013494329 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4720033890 con 0x7f472c0ff420 2024-06-18T02:57:47.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:47.510+0000 7f4734309640 1 -- 172.21.15.16:0/2013494329 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f472c0fbbf0 con 0x7f472c0ff420 2024-06-18T02:57:47.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:47.511+0000 7f4734309640 1 -- 172.21.15.16:0/2013494329 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f472c0fa630 con 0x7f472c0ff420 2024-06-18T02:57:47.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:47.512+0000 7f471effd640 1 -- 172.21.15.16:0/2013494329 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f4720030020 con 0x7f472c0ff420 2024-06-18T02:57:47.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:47.512+0000 7f471effd640 1 --2- 172.21.15.16:0/2013494329 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f4718065430 0x7f47180678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:57:47.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:47.512+0000 7f471effd640 1 -- 172.21.15.16:0/2013494329 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f47200a2200 con 0x7f472c0ff420 2024-06-18T02:57:47.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:47.512+0000 7f47320a5640 1 --2- 172.21.15.16:0/2013494329 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f4718065430 0x7f47180678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:57:47.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:47.513+0000 7f47320a5640 1 --2- 172.21.15.16:0/2013494329 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f4718065430 0x7f47180678e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f472c0fc6a0 tx=0x7f472800b040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:57:47.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:47.517+0000 7f471effd640 1 -- 172.21.15.16:0/2013494329 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f47200707e0 con 0x7f472c0ff420 2024-06-18T02:57:47.754 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:47.753+0000 7f4734309640 1 -- 172.21.15.16:0/2013494329 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f472c104ec0 con 0x7f472c0ff420 2024-06-18T02:57:47.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:47.757+0000 7f471effd640 1 -- 172.21.15.16:0/2013494329 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v14) v1 ==== 170+0+59 (secure 0 0 0) 0x7f4720073e00 con 0x7f472c0ff420 2024-06-18T02:57:47.757 INFO:teuthology.orchestra.run.smithi016.stdout:[client.0] 2024-06-18T02:57:47.758 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQCr93Bm08YELRAAkvYx/iHL7k1D7T/OkKI0pQ== 2024-06-18T02:57:47.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:47.759+0000 7f4734309640 1 -- 172.21.15.16:0/2013494329 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f4718065430 msgr2=0x7f47180678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:57:47.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:47.759+0000 7f4734309640 1 --2- 172.21.15.16:0/2013494329 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f4718065430 0x7f47180678e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f472c0fc6a0 tx=0x7f472800b040 comp rx=0 tx=0).stop 2024-06-18T02:57:47.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:47.759+0000 7f4734309640 1 -- 172.21.15.16:0/2013494329 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f472c0ff420 msgr2=0x7f472c0fade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:57:47.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:47.759+0000 7f4734309640 1 --2- 172.21.15.16:0/2013494329 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f472c0ff420 0x7f472c0fade0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f4720009f80 tx=0x7f4720009da0 comp rx=0 tx=0).stop 2024-06-18T02:57:47.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:47.759+0000 7f4734309640 1 -- 172.21.15.16:0/2013494329 shutdown_connections 2024-06-18T02:57:47.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:47.759+0000 7f4734309640 1 --2- 172.21.15.16:0/2013494329 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f4718065430 0x7f47180678e0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:57:47.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:47.759+0000 7f4734309640 1 --2- 172.21.15.16:0/2013494329 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f472c0ff420 0x7f472c0fade0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:57:47.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:47.759+0000 7f4734309640 1 --2- 172.21.15.16:0/2013494329 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f472c066410 0x7f472c0fc8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:57:47.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:47.759+0000 7f4734309640 1 -- 172.21.15.16:0/2013494329 >> 172.21.15.16:0/2013494329 conn(0x7f472c0f6990 msgr2=0x7f472c0f6db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:57:47.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:47.759+0000 7f4734309640 1 -- 172.21.15.16:0/2013494329 shutdown_connections 2024-06-18T02:57:47.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:47.759+0000 7f4734309640 1 -- 172.21.15.16:0/2013494329 wait complete. 2024-06-18T02:57:48.466 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-06-18T02:57:48.467 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-06-18T02:57:48.467 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-06-18T02:57:48.509 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-06-18T02:57:48.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:48 smithi123 ceph-mon[36358]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:57:48.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:48 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/2013494329' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-18T02:57:48.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:48 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/2013494329' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-18T02:57:48.776 INFO:teuthology.orchestra.run.smithi123.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi123/config 2024-06-18T02:57:48.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:48 smithi016 ceph-mon[29214]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:57:48.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:48 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/2013494329' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-18T02:57:48.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:48 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/2013494329' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-18T02:57:50.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:50.297+0000 7f668c3c1640 1 -- 172.21.15.123:0/4276259236 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66840fc490 msgr2=0x7f66840fc900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:57:50.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:50.297+0000 7f668c3c1640 1 --2- 172.21.15.123:0/4276259236 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66840fc490 0x7f66840fc900 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6678009c50 tx=0x7f6678004d10 comp rx=0 tx=0).stop 2024-06-18T02:57:50.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:50.297+0000 7f668c3c1640 1 -- 172.21.15.123:0/4276259236 shutdown_connections 2024-06-18T02:57:50.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:50.297+0000 7f668c3c1640 1 --2- 172.21.15.123:0/4276259236 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66840fc490 0x7f66840fc900 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:57:50.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:50.297+0000 7f668c3c1640 1 --2- 172.21.15.123:0/4276259236 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66840fb1c0 0x7f66840fb5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:57:50.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:50.297+0000 7f668c3c1640 1 -- 172.21.15.123:0/4276259236 >> 172.21.15.123:0/4276259236 conn(0x7f66840f6980 msgr2=0x7f66840f8db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:57:50.299 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:50.298+0000 7f668c3c1640 1 -- 172.21.15.123:0/4276259236 shutdown_connections 2024-06-18T02:57:50.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:50.298+0000 7f668c3c1640 1 -- 172.21.15.123:0/4276259236 wait complete. 2024-06-18T02:57:50.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:50.298+0000 7f668c3c1640 1 Processor -- start 2024-06-18T02:57:50.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:50.298+0000 7f668c3c1640 1 -- start start 2024-06-18T02:57:50.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:50.298+0000 7f668c3c1640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66840fb1c0 0x7f66840ff2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:57:50.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:50.298+0000 7f668c3c1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66840fc490 0x7f66840fd7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:57:50.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:50.298+0000 7f668c3c1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66840fddf0 con 0x7f66840fc490 2024-06-18T02:57:50.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:50.298+0000 7f668c3c1640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66840fdf30 con 0x7f66840fb1c0 2024-06-18T02:57:50.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:50.298+0000 7f668a15d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66840fb1c0 0x7f66840ff2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:57:50.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:50.298+0000 7f668a15d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66840fb1c0 0x7f66840ff2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:47500/0 (socket says 172.21.15.123:47500) 2024-06-18T02:57:50.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:50.298+0000 7f668a15d640 1 -- 172.21.15.123:0/3308092626 learned_addr learned my addr 172.21.15.123:0/3308092626 (peer_addr_for_me v2:172.21.15.123:0/0) 2024-06-18T02:57:50.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:50.298+0000 7f668995c640 1 --2- 172.21.15.123:0/3308092626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66840fc490 0x7f66840fd7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:57:50.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:50.299+0000 7f668a15d640 1 -- 172.21.15.123:0/3308092626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66840fc490 msgr2=0x7f66840fd7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:57:50.300 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:50.299+0000 7f668a15d640 1 --2- 172.21.15.123:0/3308092626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66840fc490 0x7f66840fd7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:57:50.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:50.299+0000 7f668a15d640 1 -- 172.21.15.123:0/3308092626 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66780098b0 con 0x7f66840fb1c0 2024-06-18T02:57:50.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:50.299+0000 7f668a15d640 1 --2- 172.21.15.123:0/3308092626 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66840fb1c0 0x7f66840ff2e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f667c00b790 tx=0x7f667c0091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:57:50.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:50.299+0000 7f6682ffd640 1 -- 172.21.15.123:0/3308092626 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f667c00c700 con 0x7f66840fb1c0 2024-06-18T02:57:50.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:50.299+0000 7f668c3c1640 1 -- 172.21.15.123:0/3308092626 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66840fe210 con 0x7f66840fb1c0 2024-06-18T02:57:50.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:50.299+0000 7f6682ffd640 1 -- 172.21.15.123:0/3308092626 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f667c00cd30 con 0x7f66840fb1c0 2024-06-18T02:57:50.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:50.299+0000 7f6682ffd640 1 -- 172.21.15.123:0/3308092626 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f667c013370 con 0x7f66840fb1c0 2024-06-18T02:57:50.301 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:50.299+0000 7f668c3c1640 1 -- 172.21.15.123:0/3308092626 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6684114fd0 con 0x7f66840fb1c0 2024-06-18T02:57:50.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:50.300+0000 7f668c3c1640 1 -- 172.21.15.123:0/3308092626 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66840fa620 con 0x7f66840fb1c0 2024-06-18T02:57:50.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:50.300+0000 7f6682ffd640 1 -- 172.21.15.123:0/3308092626 <== mon.1 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f667c012070 con 0x7f66840fb1c0 2024-06-18T02:57:50.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:50.301+0000 7f6682ffd640 1 --2- 172.21.15.123:0/3308092626 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f666c065430 0x7f666c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:57:50.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:50.301+0000 7f6682ffd640 1 -- 172.21.15.123:0/3308092626 <== mon.1 v2:172.21.15.123:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f667c0820b0 con 0x7f66840fb1c0 2024-06-18T02:57:50.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:50.301+0000 7f668995c640 1 --2- 172.21.15.123:0/3308092626 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f666c065430 0x7f666c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:57:50.302 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:50.302+0000 7f668995c640 1 --2- 172.21.15.123:0/3308092626 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f666c065430 0x7f666c0678e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6678000f80 tx=0x7f6678004d10 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:57:50.305 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:50.305+0000 7f6682ffd640 1 -- 172.21.15.123:0/3308092626 <== mon.1 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f667c050560 con 0x7f66840fb1c0 2024-06-18T02:57:50.532 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:50.532+0000 7f668c3c1640 1 -- 172.21.15.123:0/3308092626 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f6684100ef0 con 0x7f66840fb1c0 2024-06-18T02:57:50.536 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:50.536+0000 7f6682ffd640 1 -- 172.21.15.123:0/3308092626 <== mon.1 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v15) v1 ==== 170+0+59 (secure 0 0 0) 0x7f6684100ef0 con 0x7f66840fb1c0 2024-06-18T02:57:50.536 INFO:teuthology.orchestra.run.smithi123.stdout:[client.1] 2024-06-18T02:57:50.536 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQCu93BmT0vPHxAAUrrHnIcrt418LKc0iQp3Yw== 2024-06-18T02:57:50.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:50.538+0000 7f668c3c1640 1 -- 172.21.15.123:0/3308092626 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f666c065430 msgr2=0x7f666c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:57:50.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:50.538+0000 7f668c3c1640 1 --2- 172.21.15.123:0/3308092626 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f666c065430 0x7f666c0678e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6678000f80 tx=0x7f6678004d10 comp rx=0 tx=0).stop 2024-06-18T02:57:50.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:50.538+0000 7f668c3c1640 1 -- 172.21.15.123:0/3308092626 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66840fb1c0 msgr2=0x7f66840ff2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:57:50.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:50.538+0000 7f668c3c1640 1 --2- 172.21.15.123:0/3308092626 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66840fb1c0 0x7f66840ff2e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f667c00b790 tx=0x7f667c0091d0 comp rx=0 tx=0).stop 2024-06-18T02:57:50.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:50.538+0000 7f668c3c1640 1 -- 172.21.15.123:0/3308092626 shutdown_connections 2024-06-18T02:57:50.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:50.538+0000 7f668c3c1640 1 --2- 172.21.15.123:0/3308092626 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f666c065430 0x7f666c0678e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:57:50.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:50.538+0000 7f668c3c1640 1 --2- 172.21.15.123:0/3308092626 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66840fc490 0x7f66840fd7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:57:50.538 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:50.538+0000 7f668c3c1640 1 --2- 172.21.15.123:0/3308092626 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66840fb1c0 0x7f66840ff2e0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:57:50.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:50.538+0000 7f668c3c1640 1 -- 172.21.15.123:0/3308092626 >> 172.21.15.123:0/3308092626 conn(0x7f66840f6980 msgr2=0x7f6684065de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:57:50.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:50.538+0000 7f668c3c1640 1 -- 172.21.15.123:0/3308092626 shutdown_connections 2024-06-18T02:57:50.539 INFO:teuthology.orchestra.run.smithi123.stderr:2024-06-18T02:57:50.538+0000 7f668c3c1640 1 -- 172.21.15.123:0/3308092626 wait complete. 2024-06-18T02:57:50.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:50 smithi016 ceph-mon[29214]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:57:50.873 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:50 smithi123 ceph-mon[36358]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:57:51.132 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2024-06-18T02:57:51.133 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-06-18T02:57:51.133 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-06-18T02:57:51.170 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-06-18T02:57:51.170 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-06-18T02:57:51.171 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph mgr dump --format=json 2024-06-18T02:57:51.448 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T02:57:51.796 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:51 smithi016 ceph-mon[29214]: from='client.? 172.21.15.123:0/3308092626' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-18T02:57:51.797 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:51 smithi016 ceph-mon[29214]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-18T02:57:51.797 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:51 smithi016 ceph-mon[29214]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-18T02:57:51.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:51 smithi123 ceph-mon[36358]: from='client.? 172.21.15.123:0/3308092626' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-18T02:57:51.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:51 smithi123 ceph-mon[36358]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-18T02:57:51.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:51 smithi123 ceph-mon[36358]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-18T02:57:52.792 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:52 smithi016 ceph-mon[29214]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:57:52.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:52 smithi123 ceph-mon[36358]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:57:53.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:53.142+0000 7f92cfb8d640 1 -- 172.21.15.16:0/2808802391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92c8101ba0 msgr2=0x7f92c80fb230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:57:53.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:53.142+0000 7f92cfb8d640 1 --2- 172.21.15.16:0/2808802391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92c8101ba0 0x7f92c80fb230 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f92b80048d0 tx=0x7f92b8009da0 comp rx=0 tx=0).stop 2024-06-18T02:57:53.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:53.142+0000 7f92cfb8d640 1 -- 172.21.15.16:0/2808802391 shutdown_connections 2024-06-18T02:57:53.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:53.142+0000 7f92cfb8d640 1 --2- 172.21.15.16:0/2808802391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92c8101ba0 0x7f92c80fb230 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:57:53.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:53.142+0000 7f92cfb8d640 1 --2- 172.21.15.16:0/2808802391 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92c8101200 0x7f92c81015d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:57:53.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:53.142+0000 7f92cfb8d640 1 -- 172.21.15.16:0/2808802391 >> 172.21.15.16:0/2808802391 conn(0x7f92c80f6df0 msgr2=0x7f92c80f9200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:57:53.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:53.143+0000 7f92cfb8d640 1 -- 172.21.15.16:0/2808802391 shutdown_connections 2024-06-18T02:57:53.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:53.143+0000 7f92cfb8d640 1 -- 172.21.15.16:0/2808802391 wait complete. 2024-06-18T02:57:53.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:53.143+0000 7f92cfb8d640 1 Processor -- start 2024-06-18T02:57:53.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:53.143+0000 7f92cfb8d640 1 -- start start 2024-06-18T02:57:53.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:53.143+0000 7f92cfb8d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92c8101200 0x7f92c8115200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:57:53.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:53.143+0000 7f92cfb8d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92c8101ba0 0x7f92c8115740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:57:53.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:53.143+0000 7f92cfb8d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92c8115d40 con 0x7f92c8101200 2024-06-18T02:57:53.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:53.143+0000 7f92cfb8d640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92c810f180 con 0x7f92c8101ba0 2024-06-18T02:57:53.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:53.143+0000 7f92cd929640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92c8101200 0x7f92c8115200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:57:53.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:53.143+0000 7f92cd929640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92c8101200 0x7f92c8115200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56636/0 (socket says 172.21.15.16:56636) 2024-06-18T02:57:53.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:53.144+0000 7f92cd929640 1 -- 172.21.15.16:0/263118404 learned_addr learned my addr 172.21.15.16:0/263118404 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:57:53.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:53.144+0000 7f92cd128640 1 --2- 172.21.15.16:0/263118404 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92c8101ba0 0x7f92c8115740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:57:53.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:53.144+0000 7f92cd929640 1 -- 172.21.15.16:0/263118404 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92c8101ba0 msgr2=0x7f92c8115740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:57:53.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:53.144+0000 7f92cd929640 1 --2- 172.21.15.16:0/263118404 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92c8101ba0 0x7f92c8115740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:57:53.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:53.144+0000 7f92cd929640 1 -- 172.21.15.16:0/263118404 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92b8005040 con 0x7f92c8101200 2024-06-18T02:57:53.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:53.144+0000 7f92cd929640 1 --2- 172.21.15.16:0/263118404 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92c8101200 0x7f92c8115200 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f92c400c830 tx=0x7f92c4004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:57:53.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:53.144+0000 7f92be7fc640 1 -- 172.21.15.16:0/263118404 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f92c400a850 con 0x7f92c8101200 2024-06-18T02:57:53.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:53.144+0000 7f92be7fc640 1 -- 172.21.15.16:0/263118404 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f92c400ae80 con 0x7f92c8101200 2024-06-18T02:57:53.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:53.144+0000 7f92cfb8d640 1 -- 172.21.15.16:0/263118404 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92c810f460 con 0x7f92c8101200 2024-06-18T02:57:53.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:53.144+0000 7f92be7fc640 1 -- 172.21.15.16:0/263118404 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f92c4012ea0 con 0x7f92c8101200 2024-06-18T02:57:53.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:53.144+0000 7f92cfb8d640 1 -- 172.21.15.16:0/263118404 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92c810fa00 con 0x7f92c8101200 2024-06-18T02:57:53.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:53.145+0000 7f92cfb8d640 1 -- 172.21.15.16:0/263118404 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92c80fc970 con 0x7f92c8101200 2024-06-18T02:57:53.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:53.146+0000 7f92be7fc640 1 -- 172.21.15.16:0/263118404 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f92c4012400 con 0x7f92c8101200 2024-06-18T02:57:53.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:53.146+0000 7f92be7fc640 1 --2- 172.21.15.16:0/263118404 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f92b4065430 0x7f92b40678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:57:53.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:53.146+0000 7f92be7fc640 1 -- 172.21.15.16:0/263118404 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f92c4082620 con 0x7f92c8101200 2024-06-18T02:57:53.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:53.146+0000 7f92cd128640 1 --2- 172.21.15.16:0/263118404 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f92b4065430 0x7f92b40678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:57:53.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:53.146+0000 7f92cd128640 1 --2- 172.21.15.16:0/263118404 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f92b4065430 0x7f92b40678e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f92c8110ac0 tx=0x7f92b8035040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:57:53.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:53.150+0000 7f92be7fc640 1 -- 172.21.15.16:0/263118404 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f92c4050cb0 con 0x7f92c8101200 2024-06-18T02:57:53.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:53.374+0000 7f92cfb8d640 1 -- 172.21.15.16:0/263118404 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f92c804b480 con 0x7f92c8101200 2024-06-18T02:57:53.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:53.377+0000 7f92be7fc640 1 -- 172.21.15.16:0/263118404 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v19) v1 ==== 74+0+161447 (secure 0 0 0) 0x7f92c40126b0 con 0x7f92c8101200 2024-06-18T02:57:53.379 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:57:53.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:53.380+0000 7f92cfb8d640 1 -- 172.21.15.16:0/263118404 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f92b4065430 msgr2=0x7f92b40678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:57:53.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:53.380+0000 7f92cfb8d640 1 --2- 172.21.15.16:0/263118404 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f92b4065430 0x7f92b40678e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f92c8110ac0 tx=0x7f92b8035040 comp rx=0 tx=0).stop 2024-06-18T02:57:53.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:53.380+0000 7f92cfb8d640 1 -- 172.21.15.16:0/263118404 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92c8101200 msgr2=0x7f92c8115200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:57:53.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:53.380+0000 7f92cfb8d640 1 --2- 172.21.15.16:0/263118404 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92c8101200 0x7f92c8115200 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f92c400c830 tx=0x7f92c4004430 comp rx=0 tx=0).stop 2024-06-18T02:57:53.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:53.380+0000 7f92cfb8d640 1 -- 172.21.15.16:0/263118404 shutdown_connections 2024-06-18T02:57:53.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:53.380+0000 7f92cfb8d640 1 --2- 172.21.15.16:0/263118404 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f92b4065430 0x7f92b40678e0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:57:53.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:53.380+0000 7f92cfb8d640 1 --2- 172.21.15.16:0/263118404 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92c8101ba0 0x7f92c8115740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:57:53.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:53.380+0000 7f92cfb8d640 1 --2- 172.21.15.16:0/263118404 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f92c8101200 0x7f92c8115200 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:57:53.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:53.380+0000 7f92cfb8d640 1 -- 172.21.15.16:0/263118404 >> 172.21.15.16:0/263118404 conn(0x7f92c80f6df0 msgr2=0x7f92c80f84d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:57:53.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:53.380+0000 7f92cfb8d640 1 -- 172.21.15.16:0/263118404 shutdown_connections 2024-06-18T02:57:53.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:53.380+0000 7f92cfb8d640 1 -- 172.21.15.16:0/263118404 wait complete. 2024-06-18T02:57:53.741 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:53 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/263118404' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-06-18T02:57:53.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:53 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/263118404' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-06-18T02:57:54.170 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":19,"flags":0,"active_gid":14216,"active_name":"smithi016.amxydz","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":3814182425},{"type":"v1","addr":"172.21.15.16:6801","nonce":3814182425}]},"active_addr":"172.21.15.16:6801/3814182425","active_change":"2024-06-18T02:54:11.194846+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":14236,"name":"smithi123.olpkyi","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":false,"error_string":"No module named 'kubernetes.client.models.v1_event'","module_options":{}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":false,"error_string":"No module named 'kubernetes.client.models.v1_event'","module_options":{}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.16:8443/","prometheus":"http://172.21.15.16:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":3004679536}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":3292584211}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":3099940395}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":2268244769}]}]} 2024-06-18T02:57:54.174 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-06-18T02:57:54.174 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-06-18T02:57:54.174 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph osd dump --format=json 2024-06-18T02:57:54.443 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T02:57:54.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:54 smithi016 ceph-mon[29214]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:57:54.986 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:54 smithi123 ceph-mon[36358]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:57:56.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:56.056+0000 7f9907db3640 1 -- 172.21.15.16:0/956644561 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f99000fac10 msgr2=0x7f99000fafe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:57:56.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:56.056+0000 7f9907db3640 1 --2- 172.21.15.16:0/956644561 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f99000fac10 0x7f99000fafe0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f98fc0048d0 tx=0x7f98fc009da0 comp rx=0 tx=0).stop 2024-06-18T02:57:56.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:56.056+0000 7f9907db3640 1 -- 172.21.15.16:0/956644561 shutdown_connections 2024-06-18T02:57:56.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:56.056+0000 7f9907db3640 1 --2- 172.21.15.16:0/956644561 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99000fb520 0x7f990010c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:57:56.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:56.056+0000 7f9907db3640 1 --2- 172.21.15.16:0/956644561 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f99000fac10 0x7f99000fafe0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:57:56.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:56.056+0000 7f9907db3640 1 -- 172.21.15.16:0/956644561 >> 172.21.15.16:0/956644561 conn(0x7f9900070a40 msgr2=0x7f9900070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:57:56.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:56.057+0000 7f9907db3640 1 -- 172.21.15.16:0/956644561 shutdown_connections 2024-06-18T02:57:56.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:56.057+0000 7f9907db3640 1 -- 172.21.15.16:0/956644561 wait complete. 2024-06-18T02:57:56.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:56.057+0000 7f9907db3640 1 Processor -- start 2024-06-18T02:57:56.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:56.057+0000 7f9907db3640 1 -- start start 2024-06-18T02:57:56.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:56.057+0000 7f9907db3640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99000fac10 0x7f9900119670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:57:56.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:56.057+0000 7f9907db3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f99000fb520 0x7f9900119bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:57:56.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:56.057+0000 7f9907db3640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f990011a2b0 con 0x7f99000fb520 2024-06-18T02:57:56.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:56.057+0000 7f9907db3640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9900113610 con 0x7f99000fac10 2024-06-18T02:57:56.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:56.058+0000 7f990534e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f99000fb520 0x7f9900119bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:57:56.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:56.058+0000 7f990534e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f99000fb520 0x7f9900119bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56646/0 (socket says 172.21.15.16:56646) 2024-06-18T02:57:56.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:56.058+0000 7f990534e640 1 -- 172.21.15.16:0/872847501 learned_addr learned my addr 172.21.15.16:0/872847501 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:57:56.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:56.058+0000 7f9905b4f640 1 --2- 172.21.15.16:0/872847501 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99000fac10 0x7f9900119670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:57:56.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:56.058+0000 7f990534e640 1 -- 172.21.15.16:0/872847501 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99000fac10 msgr2=0x7f9900119670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:57:56.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:56.058+0000 7f990534e640 1 --2- 172.21.15.16:0/872847501 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99000fac10 0x7f9900119670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:57:56.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:56.058+0000 7f990534e640 1 -- 172.21.15.16:0/872847501 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98fc005040 con 0x7f99000fb520 2024-06-18T02:57:56.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:56.058+0000 7f990534e640 1 --2- 172.21.15.16:0/872847501 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f99000fb520 0x7f9900119bd0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f98f000b790 tx=0x7f98f00091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:57:56.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:56.058+0000 7f98f67fc640 1 -- 172.21.15.16:0/872847501 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f98f000c700 con 0x7f99000fb520 2024-06-18T02:57:56.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:56.058+0000 7f9907db3640 1 -- 172.21.15.16:0/872847501 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99001138f0 con 0x7f99000fb520 2024-06-18T02:57:56.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:56.058+0000 7f98f67fc640 1 -- 172.21.15.16:0/872847501 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f98f000cd30 con 0x7f99000fb520 2024-06-18T02:57:56.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:56.058+0000 7f98f67fc640 1 -- 172.21.15.16:0/872847501 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f98f0013370 con 0x7f99000fb520 2024-06-18T02:57:56.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:56.058+0000 7f9907db3640 1 -- 172.21.15.16:0/872847501 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9900113de0 con 0x7f99000fb520 2024-06-18T02:57:56.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:56.059+0000 7f9907db3640 1 -- 172.21.15.16:0/872847501 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99000fb9f0 con 0x7f99000fb520 2024-06-18T02:57:56.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:56.060+0000 7f98f67fc640 1 -- 172.21.15.16:0/872847501 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f98f0012070 con 0x7f99000fb520 2024-06-18T02:57:56.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:56.060+0000 7f98f67fc640 1 --2- 172.21.15.16:0/872847501 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f98ec065430 0x7f98ec0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:57:56.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:56.061+0000 7f9905b4f640 1 --2- 172.21.15.16:0/872847501 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f98ec065430 0x7f98ec0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:57:56.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:56.061+0000 7f98f67fc640 1 -- 172.21.15.16:0/872847501 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f98f0081e70 con 0x7f99000fb520 2024-06-18T02:57:56.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:56.061+0000 7f9905b4f640 1 --2- 172.21.15.16:0/872847501 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f98ec065430 0x7f98ec0678e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f98fc000c00 tx=0x7f98fc035040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:57:56.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:56.065+0000 7f98f67fc640 1 -- 172.21.15.16:0/872847501 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f98f0050280 con 0x7f99000fb520 2024-06-18T02:57:56.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:56.241+0000 7f9907db3640 1 -- 172.21.15.16:0/872847501 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f990010fa50 con 0x7f99000fb520 2024-06-18T02:57:56.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:56.242+0000 7f98f67fc640 1 -- 172.21.15.16:0/872847501 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+11083 (secure 0 0 0) 0x7f98f00538a0 con 0x7f99000fb520 2024-06-18T02:57:56.242 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:57:56.243 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":41,"fsid":"9e93709a-2d1d-11ef-bc9d-c7b262605968","created":"2024-06-18T02:51:57.320739+0000","modified":"2024-06-18T02:57:36.450020+0000","last_up_change":"2024-06-18T02:57:35.446077+0000","last_in_change":"2024-06-18T02:57:16.677249+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-18T02:56:23.704246+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"75595aa3-a41d-4081-9e6a-e564de1b58ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":3634864038},{"type":"v1","addr":"172.21.15.16:6803","nonce":3634864038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":3634864038},{"type":"v1","addr":"172.21.15.16:6805","nonce":3634864038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":3634864038},{"type":"v1","addr":"172.21.15.16:6809","nonce":3634864038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":3634864038},{"type":"v1","addr":"172.21.15.16:6807","nonce":3634864038}]},"public_addr":"172.21.15.16:6803/3634864038","cluster_addr":"172.21.15.16:6805/3634864038","heartbeat_back_addr":"172.21.15.16:6809/3634864038","heartbeat_front_addr":"172.21.15.16:6807/3634864038","state":["exists","up"]},{"osd":1,"uuid":"33accf47-d8b6-4b00-b579-2897f47e381d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":500143512},{"type":"v1","addr":"172.21.15.16:6811","nonce":500143512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":500143512},{"type":"v1","addr":"172.21.15.16:6813","nonce":500143512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":500143512},{"type":"v1","addr":"172.21.15.16:6817","nonce":500143512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":500143512},{"type":"v1","addr":"172.21.15.16:6815","nonce":500143512}]},"public_addr":"172.21.15.16:6811/500143512","cluster_addr":"172.21.15.16:6813/500143512","heartbeat_back_addr":"172.21.15.16:6817/500143512","heartbeat_front_addr":"172.21.15.16:6815/500143512","state":["exists","up"]},{"osd":2,"uuid":"f9d60224-3bc7-458c-a8b7-8384a7e219d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1319853338},{"type":"v1","addr":"172.21.15.16:6819","nonce":1319853338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":1319853338},{"type":"v1","addr":"172.21.15.16:6821","nonce":1319853338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":1319853338},{"type":"v1","addr":"172.21.15.16:6825","nonce":1319853338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":1319853338},{"type":"v1","addr":"172.21.15.16:6823","nonce":1319853338}]},"public_addr":"172.21.15.16:6819/1319853338","cluster_addr":"172.21.15.16:6821/1319853338","heartbeat_back_addr":"172.21.15.16:6825/1319853338","heartbeat_front_addr":"172.21.15.16:6823/1319853338","state":["exists","up"]},{"osd":3,"uuid":"d432cb55-356d-44f1-b6f0-cc98fbec6809","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":2187102998},{"type":"v1","addr":"172.21.15.123:6801","nonce":2187102998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":2187102998},{"type":"v1","addr":"172.21.15.123:6803","nonce":2187102998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":2187102998},{"type":"v1","addr":"172.21.15.123:6807","nonce":2187102998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":2187102998},{"type":"v1","addr":"172.21.15.123:6805","nonce":2187102998}]},"public_addr":"172.21.15.123:6801/2187102998","cluster_addr":"172.21.15.123:6803/2187102998","heartbeat_back_addr":"172.21.15.123:6807/2187102998","heartbeat_front_addr":"172.21.15.123:6805/2187102998","state":["exists","up"]},{"osd":4,"uuid":"05cf06af-331b-408e-a7b4-1ae1486c74a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":4038205871},{"type":"v1","addr":"172.21.15.123:6809","nonce":4038205871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":4038205871},{"type":"v1","addr":"172.21.15.123:6811","nonce":4038205871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":4038205871},{"type":"v1","addr":"172.21.15.123:6815","nonce":4038205871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":4038205871},{"type":"v1","addr":"172.21.15.123:6813","nonce":4038205871}]},"public_addr":"172.21.15.123:6809/4038205871","cluster_addr":"172.21.15.123:6811/4038205871","heartbeat_back_addr":"172.21.15.123:6815/4038205871","heartbeat_front_addr":"172.21.15.123:6813/4038205871","state":["exists","up"]},{"osd":5,"uuid":"faa63f45-ee41-485f-afad-9762cbee8f82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4116323333},{"type":"v1","addr":"172.21.15.123:6817","nonce":4116323333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4116323333},{"type":"v1","addr":"172.21.15.123:6819","nonce":4116323333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4116323333},{"type":"v1","addr":"172.21.15.123:6823","nonce":4116323333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4116323333},{"type":"v1","addr":"172.21.15.123:6821","nonce":4116323333}]},"public_addr":"172.21.15.123:6817/4116323333","cluster_addr":"172.21.15.123:6819/4116323333","heartbeat_back_addr":"172.21.15.123:6823/4116323333","heartbeat_front_addr":"172.21.15.123:6821/4116323333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-18T02:55:25.559702+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-18T02:55:51.840436+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-18T02:56:19.314038+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-18T02:56:42.662054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-18T02:57:09.162120+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-18T02:57:33.526602+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/684280009":"2024-06-19T02:54:11.194734+0000","172.21.15.16:0/1097020524":"2024-06-19T02:52:27.324621+0000","172.21.15.16:6800/1678074233":"2024-06-19T02:52:27.324621+0000","172.21.15.16:0/1691231433":"2024-06-19T02:52:27.324621+0000","172.21.15.16:6801/1678074233":"2024-06-19T02:52:27.324621+0000","172.21.15.16:0/904626067":"2024-06-19T02:52:58.168970+0000","172.21.15.16:6800/1807632468":"2024-06-19T02:52:58.168970+0000","172.21.15.16:0/4088735442":"2024-06-19T02:54:11.194734+0000","172.21.15.16:0/1001205060":"2024-06-19T02:52:27.324621+0000","172.21.15.16:0/3691774959":"2024-06-19T02:52:58.168970+0000","172.21.15.16:6801/1807632468":"2024-06-19T02:52:58.168970+0000","172.21.15.16:0/3097100123":"2024-06-19T02:52:58.168970+0000","172.21.15.16:0/2121279768":"2024-06-19T02:54:11.194734+0000","172.21.15.16:6800/3749129160":"2024-06-19T02:54:11.194734+0000","172.21.15.16:6801/3749129160":"2024-06-19T02:54:11.194734+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-18T02:57:56.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:56.244+0000 7f9907db3640 1 -- 172.21.15.16:0/872847501 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f98ec065430 msgr2=0x7f98ec0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:57:56.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:56.244+0000 7f9907db3640 1 --2- 172.21.15.16:0/872847501 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f98ec065430 0x7f98ec0678e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f98fc000c00 tx=0x7f98fc035040 comp rx=0 tx=0).stop 2024-06-18T02:57:56.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:56.244+0000 7f9907db3640 1 -- 172.21.15.16:0/872847501 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f99000fb520 msgr2=0x7f9900119bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:57:56.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:56.244+0000 7f9907db3640 1 --2- 172.21.15.16:0/872847501 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f99000fb520 0x7f9900119bd0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f98f000b790 tx=0x7f98f00091d0 comp rx=0 tx=0).stop 2024-06-18T02:57:56.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:56.244+0000 7f9907db3640 1 -- 172.21.15.16:0/872847501 shutdown_connections 2024-06-18T02:57:56.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:56.244+0000 7f9907db3640 1 --2- 172.21.15.16:0/872847501 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f98ec065430 0x7f98ec0678e0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:57:56.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:56.244+0000 7f9907db3640 1 --2- 172.21.15.16:0/872847501 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f99000fb520 0x7f9900119bd0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:57:56.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:56.244+0000 7f9907db3640 1 --2- 172.21.15.16:0/872847501 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99000fac10 0x7f9900119670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:57:56.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:56.244+0000 7f9907db3640 1 -- 172.21.15.16:0/872847501 >> 172.21.15.16:0/872847501 conn(0x7f9900070a40 msgr2=0x7f99000fbec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:57:56.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:56.244+0000 7f9907db3640 1 -- 172.21.15.16:0/872847501 shutdown_connections 2024-06-18T02:57:56.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:56.244+0000 7f9907db3640 1 -- 172.21.15.16:0/872847501 wait complete. 2024-06-18T02:57:56.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:56 smithi016 ceph-mon[29214]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:57:56.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:56 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/872847501' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-18T02:57:56.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:56 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:57:56.843 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-06-18T02:57:56.843 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph osd dump --format=json 2024-06-18T02:57:56.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:56 smithi123 ceph-mon[36358]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:57:56.985 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:56 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/872847501' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-18T02:57:56.985 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:56 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:57:57.112 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T02:57:58.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:58.682+0000 7f8b24899640 1 -- 172.21.15.16:0/4186320881 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b200fa9e0 msgr2=0x7f8b200fae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:57:58.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:58.682+0000 7f8b24899640 1 --2- 172.21.15.16:0/4186320881 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b200fa9e0 0x7f8b200fae50 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f8b080048d0 tx=0x7f8b08009da0 comp rx=0 tx=0).stop 2024-06-18T02:57:58.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:58.682+0000 7f8b24899640 1 -- 172.21.15.16:0/4186320881 shutdown_connections 2024-06-18T02:57:58.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:58.682+0000 7f8b24899640 1 --2- 172.21.15.16:0/4186320881 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b200fa9e0 0x7f8b200fae50 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:57:58.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:58.682+0000 7f8b24899640 1 --2- 172.21.15.16:0/4186320881 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b20102370 0x7f8b20102740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:57:58.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:58.682+0000 7f8b24899640 1 -- 172.21.15.16:0/4186320881 >> 172.21.15.16:0/4186320881 conn(0x7f8b20070a40 msgr2=0x7f8b20070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:57:58.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:58.682+0000 7f8b24899640 1 -- 172.21.15.16:0/4186320881 shutdown_connections 2024-06-18T02:57:58.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:58.682+0000 7f8b24899640 1 -- 172.21.15.16:0/4186320881 wait complete. 2024-06-18T02:57:58.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:58.682+0000 7f8b24899640 1 Processor -- start 2024-06-18T02:57:58.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:58.683+0000 7f8b24899640 1 -- start start 2024-06-18T02:57:58.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:58.683+0000 7f8b24899640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b200fa9e0 0x7f8b20113070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:57:58.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:58.683+0000 7f8b24899640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b20102370 0x7f8b201135b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:57:58.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:58.683+0000 7f8b24899640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b20113bb0 con 0x7f8b200fa9e0 2024-06-18T02:57:58.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:58.683+0000 7f8b24899640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b2010cff0 con 0x7f8b20102370 2024-06-18T02:57:58.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:58.683+0000 7f8b1e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b200fa9e0 0x7f8b20113070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:57:58.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:58.683+0000 7f8b1e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b200fa9e0 0x7f8b20113070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56664/0 (socket says 172.21.15.16:56664) 2024-06-18T02:57:58.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:58.683+0000 7f8b1e59c640 1 -- 172.21.15.16:0/21722232 learned_addr learned my addr 172.21.15.16:0/21722232 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:57:58.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:58.683+0000 7f8b17fff640 1 --2- 172.21.15.16:0/21722232 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b20102370 0x7f8b201135b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:57:58.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:58.683+0000 7f8b1e59c640 1 -- 172.21.15.16:0/21722232 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b20102370 msgr2=0x7f8b201135b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:57:58.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:58.683+0000 7f8b1e59c640 1 --2- 172.21.15.16:0/21722232 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b20102370 0x7f8b201135b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:57:58.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:58.683+0000 7f8b1e59c640 1 -- 172.21.15.16:0/21722232 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b08005040 con 0x7f8b200fa9e0 2024-06-18T02:57:58.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:58.683+0000 7f8b1e59c640 1 --2- 172.21.15.16:0/21722232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b200fa9e0 0x7f8b20113070 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f8b1000c790 tx=0x7f8b100043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:57:58.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:58.684+0000 7f8b16ffd640 1 -- 172.21.15.16:0/21722232 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b1000a7a0 con 0x7f8b200fa9e0 2024-06-18T02:57:58.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:58.684+0000 7f8b24899640 1 -- 172.21.15.16:0/21722232 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b2010d2d0 con 0x7f8b200fa9e0 2024-06-18T02:57:58.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:58.684+0000 7f8b16ffd640 1 -- 172.21.15.16:0/21722232 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8b1000add0 con 0x7f8b200fa9e0 2024-06-18T02:57:58.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:58.684+0000 7f8b16ffd640 1 -- 172.21.15.16:0/21722232 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b100115e0 con 0x7f8b200fa9e0 2024-06-18T02:57:58.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:58.684+0000 7f8b24899640 1 -- 172.21.15.16:0/21722232 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b2010d7c0 con 0x7f8b200fa9e0 2024-06-18T02:57:58.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:58.685+0000 7f8b24899640 1 -- 172.21.15.16:0/21722232 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b20065530 con 0x7f8b200fa9e0 2024-06-18T02:57:58.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:58.685+0000 7f8b16ffd640 1 -- 172.21.15.16:0/21722232 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f8b10012400 con 0x7f8b200fa9e0 2024-06-18T02:57:58.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:58.685+0000 7f8b16ffd640 1 --2- 172.21.15.16:0/21722232 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f8b0c065430 0x7f8b0c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:57:58.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:58.686+0000 7f8b16ffd640 1 -- 172.21.15.16:0/21722232 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f8b10082530 con 0x7f8b200fa9e0 2024-06-18T02:57:58.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:58.686+0000 7f8b17fff640 1 --2- 172.21.15.16:0/21722232 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f8b0c065430 0x7f8b0c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:57:58.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:58.686+0000 7f8b17fff640 1 --2- 172.21.15.16:0/21722232 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f8b0c065430 0x7f8b0c0678e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8b08009f80 tx=0x7f8b08035040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:57:58.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:58.690+0000 7f8b16ffd640 1 -- 172.21.15.16:0/21722232 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f8b10050b20 con 0x7f8b200fa9e0 2024-06-18T02:57:58.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:58 smithi016 ceph-mon[29214]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:57:58.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:58.866+0000 7f8b24899640 1 -- 172.21.15.16:0/21722232 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8b2004b480 con 0x7f8b200fa9e0 2024-06-18T02:57:58.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:58.866+0000 7f8b16ffd640 1 -- 172.21.15.16:0/21722232 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+11083 (secure 0 0 0) 0x7f8b10054140 con 0x7f8b200fa9e0 2024-06-18T02:57:58.867 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:57:58.867 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":41,"fsid":"9e93709a-2d1d-11ef-bc9d-c7b262605968","created":"2024-06-18T02:51:57.320739+0000","modified":"2024-06-18T02:57:36.450020+0000","last_up_change":"2024-06-18T02:57:35.446077+0000","last_in_change":"2024-06-18T02:57:16.677249+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-18T02:56:23.704246+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"75595aa3-a41d-4081-9e6a-e564de1b58ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":3634864038},{"type":"v1","addr":"172.21.15.16:6803","nonce":3634864038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":3634864038},{"type":"v1","addr":"172.21.15.16:6805","nonce":3634864038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":3634864038},{"type":"v1","addr":"172.21.15.16:6809","nonce":3634864038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":3634864038},{"type":"v1","addr":"172.21.15.16:6807","nonce":3634864038}]},"public_addr":"172.21.15.16:6803/3634864038","cluster_addr":"172.21.15.16:6805/3634864038","heartbeat_back_addr":"172.21.15.16:6809/3634864038","heartbeat_front_addr":"172.21.15.16:6807/3634864038","state":["exists","up"]},{"osd":1,"uuid":"33accf47-d8b6-4b00-b579-2897f47e381d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":500143512},{"type":"v1","addr":"172.21.15.16:6811","nonce":500143512}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":500143512},{"type":"v1","addr":"172.21.15.16:6813","nonce":500143512}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":500143512},{"type":"v1","addr":"172.21.15.16:6817","nonce":500143512}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":500143512},{"type":"v1","addr":"172.21.15.16:6815","nonce":500143512}]},"public_addr":"172.21.15.16:6811/500143512","cluster_addr":"172.21.15.16:6813/500143512","heartbeat_back_addr":"172.21.15.16:6817/500143512","heartbeat_front_addr":"172.21.15.16:6815/500143512","state":["exists","up"]},{"osd":2,"uuid":"f9d60224-3bc7-458c-a8b7-8384a7e219d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":1319853338},{"type":"v1","addr":"172.21.15.16:6819","nonce":1319853338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":1319853338},{"type":"v1","addr":"172.21.15.16:6821","nonce":1319853338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":1319853338},{"type":"v1","addr":"172.21.15.16:6825","nonce":1319853338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":1319853338},{"type":"v1","addr":"172.21.15.16:6823","nonce":1319853338}]},"public_addr":"172.21.15.16:6819/1319853338","cluster_addr":"172.21.15.16:6821/1319853338","heartbeat_back_addr":"172.21.15.16:6825/1319853338","heartbeat_front_addr":"172.21.15.16:6823/1319853338","state":["exists","up"]},{"osd":3,"uuid":"d432cb55-356d-44f1-b6f0-cc98fbec6809","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":2187102998},{"type":"v1","addr":"172.21.15.123:6801","nonce":2187102998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":2187102998},{"type":"v1","addr":"172.21.15.123:6803","nonce":2187102998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":2187102998},{"type":"v1","addr":"172.21.15.123:6807","nonce":2187102998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6804","nonce":2187102998},{"type":"v1","addr":"172.21.15.123:6805","nonce":2187102998}]},"public_addr":"172.21.15.123:6801/2187102998","cluster_addr":"172.21.15.123:6803/2187102998","heartbeat_back_addr":"172.21.15.123:6807/2187102998","heartbeat_front_addr":"172.21.15.123:6805/2187102998","state":["exists","up"]},{"osd":4,"uuid":"05cf06af-331b-408e-a7b4-1ae1486c74a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6808","nonce":4038205871},{"type":"v1","addr":"172.21.15.123:6809","nonce":4038205871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":4038205871},{"type":"v1","addr":"172.21.15.123:6811","nonce":4038205871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":4038205871},{"type":"v1","addr":"172.21.15.123:6815","nonce":4038205871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6812","nonce":4038205871},{"type":"v1","addr":"172.21.15.123:6813","nonce":4038205871}]},"public_addr":"172.21.15.123:6809/4038205871","cluster_addr":"172.21.15.123:6811/4038205871","heartbeat_back_addr":"172.21.15.123:6815/4038205871","heartbeat_front_addr":"172.21.15.123:6813/4038205871","state":["exists","up"]},{"osd":5,"uuid":"faa63f45-ee41-485f-afad-9762cbee8f82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":4116323333},{"type":"v1","addr":"172.21.15.123:6817","nonce":4116323333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":4116323333},{"type":"v1","addr":"172.21.15.123:6819","nonce":4116323333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":4116323333},{"type":"v1","addr":"172.21.15.123:6823","nonce":4116323333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":4116323333},{"type":"v1","addr":"172.21.15.123:6821","nonce":4116323333}]},"public_addr":"172.21.15.123:6817/4116323333","cluster_addr":"172.21.15.123:6819/4116323333","heartbeat_back_addr":"172.21.15.123:6823/4116323333","heartbeat_front_addr":"172.21.15.123:6821/4116323333","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-18T02:55:25.559702+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-18T02:55:51.840436+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-18T02:56:19.314038+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-18T02:56:42.662054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-18T02:57:09.162120+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-18T02:57:33.526602+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/684280009":"2024-06-19T02:54:11.194734+0000","172.21.15.16:0/1097020524":"2024-06-19T02:52:27.324621+0000","172.21.15.16:6800/1678074233":"2024-06-19T02:52:27.324621+0000","172.21.15.16:0/1691231433":"2024-06-19T02:52:27.324621+0000","172.21.15.16:6801/1678074233":"2024-06-19T02:52:27.324621+0000","172.21.15.16:0/904626067":"2024-06-19T02:52:58.168970+0000","172.21.15.16:6800/1807632468":"2024-06-19T02:52:58.168970+0000","172.21.15.16:0/4088735442":"2024-06-19T02:54:11.194734+0000","172.21.15.16:0/1001205060":"2024-06-19T02:52:27.324621+0000","172.21.15.16:0/3691774959":"2024-06-19T02:52:58.168970+0000","172.21.15.16:6801/1807632468":"2024-06-19T02:52:58.168970+0000","172.21.15.16:0/3097100123":"2024-06-19T02:52:58.168970+0000","172.21.15.16:0/2121279768":"2024-06-19T02:54:11.194734+0000","172.21.15.16:6800/3749129160":"2024-06-19T02:54:11.194734+0000","172.21.15.16:6801/3749129160":"2024-06-19T02:54:11.194734+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-18T02:57:58.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:58.868+0000 7f8b24899640 1 -- 172.21.15.16:0/21722232 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f8b0c065430 msgr2=0x7f8b0c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:57:58.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:58.868+0000 7f8b24899640 1 --2- 172.21.15.16:0/21722232 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f8b0c065430 0x7f8b0c0678e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8b08009f80 tx=0x7f8b08035040 comp rx=0 tx=0).stop 2024-06-18T02:57:58.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:58.868+0000 7f8b24899640 1 -- 172.21.15.16:0/21722232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b200fa9e0 msgr2=0x7f8b20113070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:57:58.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:58.868+0000 7f8b24899640 1 --2- 172.21.15.16:0/21722232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b200fa9e0 0x7f8b20113070 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f8b1000c790 tx=0x7f8b100043c0 comp rx=0 tx=0).stop 2024-06-18T02:57:58.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:58.868+0000 7f8b24899640 1 -- 172.21.15.16:0/21722232 shutdown_connections 2024-06-18T02:57:58.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:58.868+0000 7f8b24899640 1 --2- 172.21.15.16:0/21722232 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f8b0c065430 0x7f8b0c0678e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:57:58.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:58.868+0000 7f8b24899640 1 --2- 172.21.15.16:0/21722232 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8b20102370 0x7f8b201135b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:57:58.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:58.868+0000 7f8b24899640 1 --2- 172.21.15.16:0/21722232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b200fa9e0 0x7f8b20113070 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:57:58.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:58.868+0000 7f8b24899640 1 -- 172.21.15.16:0/21722232 >> 172.21.15.16:0/21722232 conn(0x7f8b20070a40 msgr2=0x7f8b200f9e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:57:58.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:58.869+0000 7f8b24899640 1 -- 172.21.15.16:0/21722232 shutdown_connections 2024-06-18T02:57:58.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:57:58.869+0000 7f8b24899640 1 -- 172.21.15.16:0/21722232 wait complete. 2024-06-18T02:57:58.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:58 smithi123 ceph-mon[36358]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:57:59.599 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-06-18T02:57:59.600 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-06-18T02:57:59.600 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-06-18T02:57:59.601 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-06-18T02:57:59.601 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-06-18T02:57:59.602 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-06-18T02:57:59.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:57:59 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/21722232' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-18T02:57:59.913 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T02:57:59.945 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T02:57:59.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:57:59 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/21722232' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-18T02:58:00.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:58:00 smithi016 ceph-mon[29214]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:58:00.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:58:00 smithi123 ceph-mon[36358]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:58:02.206 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T02:58:02.207 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T02:58:02.207 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T02:58:02.208 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T02:58:02.765 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:58:02 smithi016 ceph-mon[29214]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:58:02.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:02.845+0000 7f6b96a0d640 1 -- 172.21.15.16:0/3017976638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b901022f0 msgr2=0x7f6b901026c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:02.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:02.845+0000 7f6b96a0d640 1 --2- 172.21.15.16:0/3017976638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b901022f0 0x7f6b901026c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f6b800048d0 tx=0x7f6b80009da0 comp rx=0 tx=0).stop 2024-06-18T02:58:02.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:02.845+0000 7f6b96a0d640 1 -- 172.21.15.16:0/3017976638 shutdown_connections 2024-06-18T02:58:02.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:02.845+0000 7f6b96a0d640 1 --2- 172.21.15.16:0/3017976638 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6b90102c90 0x7f6b900fa9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:02.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:02.845+0000 7f6b96a0d640 1 --2- 172.21.15.16:0/3017976638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b901022f0 0x7f6b901026c0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:02.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:02.845+0000 7f6b96a0d640 1 -- 172.21.15.16:0/3017976638 >> 172.21.15.16:0/3017976638 conn(0x7f6b90070a40 msgr2=0x7f6b90070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:58:02.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:02.845+0000 7f6b96a0d640 1 -- 172.21.15.16:0/3017976638 shutdown_connections 2024-06-18T02:58:02.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:02.845+0000 7f6b96a0d640 1 -- 172.21.15.16:0/3017976638 wait complete. 2024-06-18T02:58:02.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:02.845+0000 7f6b96a0d640 1 Processor -- start 2024-06-18T02:58:02.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:02.846+0000 7f6b96a0d640 1 -- start start 2024-06-18T02:58:02.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:02.846+0000 7f6b96a0d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6b901022f0 0x7f6b90113100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:58:02.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:02.846+0000 7f6b96a0d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b90102c90 0x7f6b90113640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:58:02.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:02.846+0000 7f6b96a0d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b90113c40 con 0x7f6b90102c90 2024-06-18T02:58:02.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:02.846+0000 7f6b96a0d640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b9010d080 con 0x7f6b901022f0 2024-06-18T02:58:02.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:02.846+0000 7f6b8f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b90102c90 0x7f6b90113640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:58:02.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:02.846+0000 7f6b8f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b90102c90 0x7f6b90113640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48550/0 (socket says 172.21.15.16:48550) 2024-06-18T02:58:02.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:02.846+0000 7f6b8f7fe640 1 -- 172.21.15.16:0/1955610007 learned_addr learned my addr 172.21.15.16:0/1955610007 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:58:02.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:02.846+0000 7f6b8ffff640 1 --2- 172.21.15.16:0/1955610007 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6b901022f0 0x7f6b90113100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:58:02.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:02.846+0000 7f6b8f7fe640 1 -- 172.21.15.16:0/1955610007 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6b901022f0 msgr2=0x7f6b90113100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:02.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:02.846+0000 7f6b8f7fe640 1 --2- 172.21.15.16:0/1955610007 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6b901022f0 0x7f6b90113100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:02.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:02.846+0000 7f6b8f7fe640 1 -- 172.21.15.16:0/1955610007 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b80005040 con 0x7f6b90102c90 2024-06-18T02:58:02.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:02.846+0000 7f6b8ffff640 1 --2- 172.21.15.16:0/1955610007 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6b901022f0 0x7f6b90113100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-18T02:58:02.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:02.846+0000 7f6b8f7fe640 1 --2- 172.21.15.16:0/1955610007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b90102c90 0x7f6b90113640 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f6b8400eb40 tx=0x7f6b8400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:58:02.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:02.847+0000 7f6b8cff9640 1 -- 172.21.15.16:0/1955610007 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6b84007c30 con 0x7f6b90102c90 2024-06-18T02:58:02.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:02.847+0000 7f6b96a0d640 1 -- 172.21.15.16:0/1955610007 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b9010d300 con 0x7f6b90102c90 2024-06-18T02:58:02.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:02.847+0000 7f6b8cff9640 1 -- 172.21.15.16:0/1955610007 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6b8400f040 con 0x7f6b90102c90 2024-06-18T02:58:02.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:02.847+0000 7f6b8cff9640 1 -- 172.21.15.16:0/1955610007 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6b84009620 con 0x7f6b90102c90 2024-06-18T02:58:02.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:02.847+0000 7f6b96a0d640 1 -- 172.21.15.16:0/1955610007 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b9010d820 con 0x7f6b90102c90 2024-06-18T02:58:02.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:02.847+0000 7f6b96a0d640 1 -- 172.21.15.16:0/1955610007 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6b9010ddb0 con 0x7f6b90102c90 2024-06-18T02:58:02.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:02.848+0000 7f6b8cff9640 1 -- 172.21.15.16:0/1955610007 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f6b84009780 con 0x7f6b90102c90 2024-06-18T02:58:02.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:02.849+0000 7f6b8cff9640 1 --2- 172.21.15.16:0/1955610007 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f6b78065430 0x7f6b780678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:58:02.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:02.849+0000 7f6b8cff9640 1 -- 172.21.15.16:0/1955610007 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f6b840820d0 con 0x7f6b90102c90 2024-06-18T02:58:02.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:02.849+0000 7f6b8ffff640 1 --2- 172.21.15.16:0/1955610007 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f6b78065430 0x7f6b780678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:58:02.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:02.849+0000 7f6b8cff9640 1 --2- 172.21.15.16:0/1955610007 >> [v2:172.21.15.16:6818/1319853338,v1:172.21.15.16:6819/1319853338] conn(0x7f6b7806b1b0 0x7f6b7806d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:58:02.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:02.849+0000 7f6b8cff9640 1 -- 172.21.15.16:0/1955610007 --> [v2:172.21.15.16:6818/1319853338,v1:172.21.15.16:6819/1319853338] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6b7806dc70 con 0x7f6b7806b1b0 2024-06-18T02:58:02.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:02.849+0000 7f6b8cff9640 1 -- 172.21.15.16:0/1955610007 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6b84084720 con 0x7f6b90102c90 2024-06-18T02:58:02.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:02.849+0000 7f6b94faa640 1 --2- 172.21.15.16:0/1955610007 >> [v2:172.21.15.16:6818/1319853338,v1:172.21.15.16:6819/1319853338] conn(0x7f6b7806b1b0 0x7f6b7806d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:58:02.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:02.849+0000 7f6b8ffff640 1 --2- 172.21.15.16:0/1955610007 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f6b78065430 0x7f6b780678e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6b80000c00 tx=0x7f6b80035040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:58:02.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:02.849+0000 7f6b94faa640 1 --2- 172.21.15.16:0/1955610007 >> [v2:172.21.15.16:6818/1319853338,v1:172.21.15.16:6819/1319853338] conn(0x7f6b7806b1b0 0x7f6b7806d5c0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:58:02.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:02.850+0000 7f6b8cff9640 1 -- 172.21.15.16:0/1955610007 <== osd.2 v2:172.21.15.16:6818/1319853338 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f6b7806dc70 con 0x7f6b7806b1b0 2024-06-18T02:58:02.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:02.863+0000 7f6b96a0d640 1 -- 172.21.15.16:0/1955610007 --> [v2:172.21.15.16:6818/1319853338,v1:172.21.15.16:6819/1319853338] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6b90106a50 con 0x7f6b7806b1b0 2024-06-18T02:58:02.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:02.863+0000 7f6b8cff9640 1 -- 172.21.15.16:0/1955610007 <== osd.2 v2:172.21.15.16:6818/1319853338 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f6b90106a50 con 0x7f6b7806b1b0 2024-06-18T02:58:02.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:02.864+0000 7f6b96a0d640 1 -- 172.21.15.16:0/1955610007 >> [v2:172.21.15.16:6818/1319853338,v1:172.21.15.16:6819/1319853338] conn(0x7f6b7806b1b0 msgr2=0x7f6b7806d5c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:02.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:02.864+0000 7f6b96a0d640 1 --2- 172.21.15.16:0/1955610007 >> [v2:172.21.15.16:6818/1319853338,v1:172.21.15.16:6819/1319853338] conn(0x7f6b7806b1b0 0x7f6b7806d5c0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:02.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:02.864+0000 7f6b96a0d640 1 -- 172.21.15.16:0/1955610007 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f6b78065430 msgr2=0x7f6b780678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:02.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:02.864+0000 7f6b96a0d640 1 --2- 172.21.15.16:0/1955610007 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f6b78065430 0x7f6b780678e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6b80000c00 tx=0x7f6b80035040 comp rx=0 tx=0).stop 2024-06-18T02:58:02.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:02.864+0000 7f6b96a0d640 1 -- 172.21.15.16:0/1955610007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b90102c90 msgr2=0x7f6b90113640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:02.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:02.864+0000 7f6b96a0d640 1 --2- 172.21.15.16:0/1955610007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b90102c90 0x7f6b90113640 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f6b8400eb40 tx=0x7f6b8400b040 comp rx=0 tx=0).stop 2024-06-18T02:58:02.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:02.864+0000 7f6b96a0d640 1 -- 172.21.15.16:0/1955610007 shutdown_connections 2024-06-18T02:58:02.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:02.864+0000 7f6b96a0d640 1 --2- 172.21.15.16:0/1955610007 >> [v2:172.21.15.16:6818/1319853338,v1:172.21.15.16:6819/1319853338] conn(0x7f6b7806b1b0 0x7f6b7806d5c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:02.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:02.864+0000 7f6b96a0d640 1 --2- 172.21.15.16:0/1955610007 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f6b78065430 0x7f6b780678e0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:02.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:02.864+0000 7f6b96a0d640 1 --2- 172.21.15.16:0/1955610007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b90102c90 0x7f6b90113640 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:02.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:02.864+0000 7f6b96a0d640 1 --2- 172.21.15.16:0/1955610007 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6b901022f0 0x7f6b90113100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:02.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:02.864+0000 7f6b96a0d640 1 -- 172.21.15.16:0/1955610007 >> 172.21.15.16:0/1955610007 conn(0x7f6b90070a40 msgr2=0x7f6b901050b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:58:02.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:02.864+0000 7f6b96a0d640 1 -- 172.21.15.16:0/1955610007 shutdown_connections 2024-06-18T02:58:02.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:02.864+0000 7f6b96a0d640 1 -- 172.21.15.16:0/1955610007 wait complete. 2024-06-18T02:58:02.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:58:02 smithi123 ceph-mon[36358]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:58:03.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:03.028+0000 7ff3d593d640 1 -- 172.21.15.16:0/2343124946 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3d0102c90 msgr2=0x7ff3d00fa9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:03.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:03.028+0000 7ff3d593d640 1 --2- 172.21.15.16:0/2343124946 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3d0102c90 0x7ff3d00fa9e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff3b80048d0 tx=0x7ff3b8009da0 comp rx=0 tx=0).stop 2024-06-18T02:58:03.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:03.028+0000 7ff3d593d640 1 -- 172.21.15.16:0/2343124946 shutdown_connections 2024-06-18T02:58:03.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:03.028+0000 7ff3d593d640 1 --2- 172.21.15.16:0/2343124946 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3d0102c90 0x7ff3d00fa9e0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:03.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:03.028+0000 7ff3d593d640 1 --2- 172.21.15.16:0/2343124946 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3d01022f0 0x7ff3d01026c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:03.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:03.028+0000 7ff3d593d640 1 -- 172.21.15.16:0/2343124946 >> 172.21.15.16:0/2343124946 conn(0x7ff3d0070a40 msgr2=0x7ff3d0070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:58:03.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:03.029+0000 7ff3d593d640 1 -- 172.21.15.16:0/2343124946 shutdown_connections 2024-06-18T02:58:03.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:03.029+0000 7ff3d593d640 1 -- 172.21.15.16:0/2343124946 wait complete. 2024-06-18T02:58:03.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:03.029+0000 7ff3d593d640 1 Processor -- start 2024-06-18T02:58:03.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:03.029+0000 7ff3d593d640 1 -- start start 2024-06-18T02:58:03.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:03.029+0000 7ff3d593d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3d01022f0 0x7ff3d01130f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:58:03.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:03.029+0000 7ff3d593d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3d0102c90 0x7ff3d0113630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:58:03.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:03.029+0000 7ff3d593d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d0113c30 con 0x7ff3d01022f0 2024-06-18T02:58:03.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:03.029+0000 7ff3d593d640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d010d070 con 0x7ff3d0102c90 2024-06-18T02:58:03.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:03.030+0000 7ff3ceffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3d01022f0 0x7ff3d01130f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:58:03.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:03.030+0000 7ff3ceffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3d01022f0 0x7ff3d01130f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48568/0 (socket says 172.21.15.16:48568) 2024-06-18T02:58:03.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:03.030+0000 7ff3ceffd640 1 -- 172.21.15.16:0/3170357123 learned_addr learned my addr 172.21.15.16:0/3170357123 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:58:03.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:03.030+0000 7ff3c67fc640 1 --2- 172.21.15.16:0/3170357123 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3d0102c90 0x7ff3d0113630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:58:03.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:03.030+0000 7ff3ceffd640 1 -- 172.21.15.16:0/3170357123 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3d0102c90 msgr2=0x7ff3d0113630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:03.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:03.030+0000 7ff3ceffd640 1 --2- 172.21.15.16:0/3170357123 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3d0102c90 0x7ff3d0113630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:03.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:03.030+0000 7ff3ceffd640 1 -- 172.21.15.16:0/3170357123 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3b8005040 con 0x7ff3d01022f0 2024-06-18T02:58:03.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:03.030+0000 7ff3c67fc640 1 --2- 172.21.15.16:0/3170357123 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3d0102c90 0x7ff3d0113630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-18T02:58:03.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:03.030+0000 7ff3ceffd640 1 --2- 172.21.15.16:0/3170357123 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3d01022f0 0x7ff3d01130f0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7ff3c000eb70 tx=0x7ff3c000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:58:03.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:03.030+0000 7ff3c7fff640 1 -- 172.21.15.16:0/3170357123 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff3c0007c30 con 0x7ff3d01022f0 2024-06-18T02:58:03.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:03.030+0000 7ff3d593d640 1 -- 172.21.15.16:0/3170357123 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3d010d350 con 0x7ff3d01022f0 2024-06-18T02:58:03.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:03.030+0000 7ff3c7fff640 1 -- 172.21.15.16:0/3170357123 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff3c000f040 con 0x7ff3d01022f0 2024-06-18T02:58:03.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:03.030+0000 7ff3c7fff640 1 -- 172.21.15.16:0/3170357123 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff3c000cc30 con 0x7ff3d01022f0 2024-06-18T02:58:03.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:03.030+0000 7ff3d593d640 1 -- 172.21.15.16:0/3170357123 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3d010d840 con 0x7ff3d01022f0 2024-06-18T02:58:03.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:03.031+0000 7ff3d593d640 1 -- 172.21.15.16:0/3170357123 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff3d010dda0 con 0x7ff3d01022f0 2024-06-18T02:58:03.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:03.032+0000 7ff3c7fff640 1 -- 172.21.15.16:0/3170357123 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7ff3c0009110 con 0x7ff3d01022f0 2024-06-18T02:58:03.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:03.032+0000 7ff3c7fff640 1 --2- 172.21.15.16:0/3170357123 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7ff3bc065430 0x7ff3bc0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:58:03.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:03.032+0000 7ff3c7fff640 1 -- 172.21.15.16:0/3170357123 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7ff3c0082710 con 0x7ff3d01022f0 2024-06-18T02:58:03.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:03.032+0000 7ff3c67fc640 1 --2- 172.21.15.16:0/3170357123 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7ff3bc065430 0x7ff3bc0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:58:03.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:03.032+0000 7ff3c7fff640 1 --2- 172.21.15.16:0/3170357123 >> [v2:172.21.15.123:6816/4116323333,v1:172.21.15.123:6817/4116323333] conn(0x7ff3bc06b1b0 0x7ff3bc06d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:58:03.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:03.032+0000 7ff3c7fff640 1 -- 172.21.15.16:0/3170357123 --> [v2:172.21.15.123:6816/4116323333,v1:172.21.15.123:6817/4116323333] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff3bc06dc70 con 0x7ff3bc06b1b0 2024-06-18T02:58:03.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:03.032+0000 7ff3c7fff640 1 -- 172.21.15.16:0/3170357123 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff3c0084d00 con 0x7ff3d01022f0 2024-06-18T02:58:03.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:03.033+0000 7ff3c67fc640 1 --2- 172.21.15.16:0/3170357123 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7ff3bc065430 0x7ff3bc0678e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff3b80048a0 tx=0x7ff3b8035040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:58:03.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:03.033+0000 7ff3cf7fe640 1 --2- 172.21.15.16:0/3170357123 >> [v2:172.21.15.123:6816/4116323333,v1:172.21.15.123:6817/4116323333] conn(0x7ff3bc06b1b0 0x7ff3bc06d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:58:03.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:03.033+0000 7ff3cf7fe640 1 --2- 172.21.15.16:0/3170357123 >> [v2:172.21.15.123:6816/4116323333,v1:172.21.15.123:6817/4116323333] conn(0x7ff3bc06b1b0 0x7ff3bc06d5c0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:58:03.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:03.034+0000 7ff3c7fff640 1 -- 172.21.15.16:0/3170357123 <== osd.5 v2:172.21.15.123:6816/4116323333 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7ff3bc06dc70 con 0x7ff3bc06b1b0 2024-06-18T02:58:03.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:03.047+0000 7ff3d593d640 1 -- 172.21.15.16:0/3170357123 --> [v2:172.21.15.123:6816/4116323333,v1:172.21.15.123:6817/4116323333] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff3d0065510 con 0x7ff3bc06b1b0 2024-06-18T02:58:03.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:03.048+0000 7ff3c7fff640 1 -- 172.21.15.16:0/3170357123 <== osd.5 v2:172.21.15.123:6816/4116323333 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7ff3d0065510 con 0x7ff3bc06b1b0 2024-06-18T02:58:03.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:03.048+0000 7ff3d593d640 1 -- 172.21.15.16:0/3170357123 >> [v2:172.21.15.123:6816/4116323333,v1:172.21.15.123:6817/4116323333] conn(0x7ff3bc06b1b0 msgr2=0x7ff3bc06d5c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:03.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:03.048+0000 7ff3d593d640 1 --2- 172.21.15.16:0/3170357123 >> [v2:172.21.15.123:6816/4116323333,v1:172.21.15.123:6817/4116323333] conn(0x7ff3bc06b1b0 0x7ff3bc06d5c0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:03.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:03.048+0000 7ff3d593d640 1 -- 172.21.15.16:0/3170357123 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7ff3bc065430 msgr2=0x7ff3bc0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:03.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:03.048+0000 7ff3d593d640 1 --2- 172.21.15.16:0/3170357123 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7ff3bc065430 0x7ff3bc0678e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff3b80048a0 tx=0x7ff3b8035040 comp rx=0 tx=0).stop 2024-06-18T02:58:03.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:03.048+0000 7ff3d593d640 1 -- 172.21.15.16:0/3170357123 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3d01022f0 msgr2=0x7ff3d01130f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:03.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:03.048+0000 7ff3d593d640 1 --2- 172.21.15.16:0/3170357123 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3d01022f0 0x7ff3d01130f0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7ff3c000eb70 tx=0x7ff3c000b040 comp rx=0 tx=0).stop 2024-06-18T02:58:03.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:03.048+0000 7ff3d593d640 1 -- 172.21.15.16:0/3170357123 shutdown_connections 2024-06-18T02:58:03.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:03.048+0000 7ff3d593d640 1 --2- 172.21.15.16:0/3170357123 >> [v2:172.21.15.123:6816/4116323333,v1:172.21.15.123:6817/4116323333] conn(0x7ff3bc06b1b0 0x7ff3bc06d5c0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:03.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:03.048+0000 7ff3d593d640 1 --2- 172.21.15.16:0/3170357123 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7ff3bc065430 0x7ff3bc0678e0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:03.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:03.048+0000 7ff3d593d640 1 --2- 172.21.15.16:0/3170357123 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3d0102c90 0x7ff3d0113630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:03.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:03.048+0000 7ff3d593d640 1 --2- 172.21.15.16:0/3170357123 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3d01022f0 0x7ff3d01130f0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:03.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:03.048+0000 7ff3d593d640 1 -- 172.21.15.16:0/3170357123 >> 172.21.15.16:0/3170357123 conn(0x7ff3d0070a40 msgr2=0x7ff3d01050b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:58:03.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:03.048+0000 7ff3d593d640 1 -- 172.21.15.16:0/3170357123 shutdown_connections 2024-06-18T02:58:03.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:03.048+0000 7ff3d593d640 1 -- 172.21.15.16:0/3170357123 wait complete. 2024-06-18T02:58:04.952 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:58:04 smithi016 ceph-mon[29214]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:58:04.983 INFO:teuthology.orchestra.run.smithi016.stdout:81604378647 2024-06-18T02:58:04.983 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-06-18T02:58:04.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:58:04 smithi123 ceph-mon[36358]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:58:05.980 INFO:teuthology.orchestra.run.smithi016.stdout:171798691849 2024-06-18T02:58:05.980 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-06-18T02:58:06.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:58:06 smithi016 ceph-mon[29214]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:58:06.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:58:06 smithi123 ceph-mon[36358]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:58:08.883 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:58:08 smithi016 ceph-mon[29214]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:58:08.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:08.882+0000 7fd2dd6ca640 1 -- 172.21.15.16:0/2420723225 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2d8102c40 msgr2=0x7fd2d80fc2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:08.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:08.882+0000 7fd2dd6ca640 1 --2- 172.21.15.16:0/2420723225 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2d8102c40 0x7fd2d80fc2d0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc0048d0 tx=0x7fd2cc009da0 comp rx=0 tx=0).stop 2024-06-18T02:58:08.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:08.882+0000 7fd2dd6ca640 1 -- 172.21.15.16:0/2420723225 shutdown_connections 2024-06-18T02:58:08.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:08.882+0000 7fd2dd6ca640 1 --2- 172.21.15.16:0/2420723225 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2d8102c40 0x7fd2d80fc2d0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:08.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:08.882+0000 7fd2dd6ca640 1 --2- 172.21.15.16:0/2420723225 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2d81022a0 0x7fd2d8102670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:08.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:08.882+0000 7fd2dd6ca640 1 -- 172.21.15.16:0/2420723225 >> 172.21.15.16:0/2420723225 conn(0x7fd2d80f7e50 msgr2=0x7fd2d80fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:58:08.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:08.882+0000 7fd2dd6ca640 1 -- 172.21.15.16:0/2420723225 shutdown_connections 2024-06-18T02:58:08.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:08.882+0000 7fd2dd6ca640 1 -- 172.21.15.16:0/2420723225 wait complete. 2024-06-18T02:58:08.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:08.883+0000 7fd2dd6ca640 1 Processor -- start 2024-06-18T02:58:08.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:08.883+0000 7fd2dd6ca640 1 -- start start 2024-06-18T02:58:08.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:08.883+0000 7fd2dd6ca640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2d81022a0 0x7fd2d8113020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:58:08.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:08.883+0000 7fd2dd6ca640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2d8102c40 0x7fd2d8113560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:58:08.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:08.883+0000 7fd2dd6ca640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d8113b60 con 0x7fd2d8102c40 2024-06-18T02:58:08.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:08.883+0000 7fd2dd6ca640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d810cfa0 con 0x7fd2d81022a0 2024-06-18T02:58:08.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:08.884+0000 7fd2d67fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2d8102c40 0x7fd2d8113560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:58:08.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:08.884+0000 7fd2d67fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2d8102c40 0x7fd2d8113560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48596/0 (socket says 172.21.15.16:48596) 2024-06-18T02:58:08.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:08.884+0000 7fd2d67fc640 1 -- 172.21.15.16:0/79553708 learned_addr learned my addr 172.21.15.16:0/79553708 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:58:08.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:08.884+0000 7fd2d6ffd640 1 --2- 172.21.15.16:0/79553708 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2d81022a0 0x7fd2d8113020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:58:08.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:08.884+0000 7fd2d67fc640 1 -- 172.21.15.16:0/79553708 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2d81022a0 msgr2=0x7fd2d8113020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:08.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:08.884+0000 7fd2d67fc640 1 --2- 172.21.15.16:0/79553708 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2d81022a0 0x7fd2d8113020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:08.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:08.884+0000 7fd2d67fc640 1 -- 172.21.15.16:0/79553708 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2cc005040 con 0x7fd2d8102c40 2024-06-18T02:58:08.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:08.884+0000 7fd2d67fc640 1 --2- 172.21.15.16:0/79553708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2d8102c40 0x7fd2d8113560 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc003080 tx=0x7fd2cc009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:58:08.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:08.884+0000 7fd2bf7fe640 1 -- 172.21.15.16:0/79553708 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd2cc038070 con 0x7fd2d8102c40 2024-06-18T02:58:08.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:08.884+0000 7fd2dd6ca640 1 -- 172.21.15.16:0/79553708 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2d810d220 con 0x7fd2d8102c40 2024-06-18T02:58:08.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:08.884+0000 7fd2bf7fe640 1 -- 172.21.15.16:0/79553708 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd2cc032970 con 0x7fd2d8102c40 2024-06-18T02:58:08.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:08.884+0000 7fd2bf7fe640 1 -- 172.21.15.16:0/79553708 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd2cc03d960 con 0x7fd2d8102c40 2024-06-18T02:58:08.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:08.884+0000 7fd2dd6ca640 1 -- 172.21.15.16:0/79553708 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2d810d760 con 0x7fd2d8102c40 2024-06-18T02:58:08.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:08.885+0000 7fd2dd6ca640 1 -- 172.21.15.16:0/79553708 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd2d810dc40 con 0x7fd2d8102c40 2024-06-18T02:58:08.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:08.886+0000 7fd2bf7fe640 1 -- 172.21.15.16:0/79553708 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fd2cc030020 con 0x7fd2d8102c40 2024-06-18T02:58:08.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:08.886+0000 7fd2bf7fe640 1 --2- 172.21.15.16:0/79553708 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fd2c0065430 0x7fd2c00678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:58:08.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:08.886+0000 7fd2bf7fe640 1 -- 172.21.15.16:0/79553708 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fd2cc0a2540 con 0x7fd2d8102c40 2024-06-18T02:58:08.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:08.886+0000 7fd2d6ffd640 1 --2- 172.21.15.16:0/79553708 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fd2c0065430 0x7fd2c00678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:58:08.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:08.886+0000 7fd2bf7fe640 1 --2- 172.21.15.16:0/79553708 >> [v2:172.21.15.123:6800/2187102998,v1:172.21.15.123:6801/2187102998] conn(0x7fd2c006b1b0 0x7fd2c006d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:58:08.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:08.886+0000 7fd2bf7fe640 1 -- 172.21.15.16:0/79553708 --> [v2:172.21.15.123:6800/2187102998,v1:172.21.15.123:6801/2187102998] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd2c006dc70 con 0x7fd2c006b1b0 2024-06-18T02:58:08.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:08.886+0000 7fd2bf7fe640 1 -- 172.21.15.16:0/79553708 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd2cc0a4b30 con 0x7fd2d8102c40 2024-06-18T02:58:08.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:08.887+0000 7fd2d6ffd640 1 --2- 172.21.15.16:0/79553708 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fd2c0065430 0x7fd2c00678e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd2c8009af0 tx=0x7fd2c800b040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:58:08.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:08.887+0000 7fd2d77fe640 1 --2- 172.21.15.16:0/79553708 >> [v2:172.21.15.123:6800/2187102998,v1:172.21.15.123:6801/2187102998] conn(0x7fd2c006b1b0 0x7fd2c006d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:58:08.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:08.887+0000 7fd2d77fe640 1 --2- 172.21.15.16:0/79553708 >> [v2:172.21.15.123:6800/2187102998,v1:172.21.15.123:6801/2187102998] conn(0x7fd2c006b1b0 0x7fd2c006d5c0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:58:08.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:08.888+0000 7fd2bf7fe640 1 -- 172.21.15.16:0/79553708 <== osd.3 v2:172.21.15.123:6800/2187102998 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fd2c006dc70 con 0x7fd2c006b1b0 2024-06-18T02:58:08.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:08.901+0000 7fd2dd6ca640 1 -- 172.21.15.16:0/79553708 --> [v2:172.21.15.123:6800/2187102998,v1:172.21.15.123:6801/2187102998] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd2d80f9840 con 0x7fd2c006b1b0 2024-06-18T02:58:08.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:08.901+0000 7fd2bf7fe640 1 -- 172.21.15.16:0/79553708 <== osd.3 v2:172.21.15.123:6800/2187102998 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fd2d80f9840 con 0x7fd2c006b1b0 2024-06-18T02:58:08.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:08.902+0000 7fd2dd6ca640 1 -- 172.21.15.16:0/79553708 >> [v2:172.21.15.123:6800/2187102998,v1:172.21.15.123:6801/2187102998] conn(0x7fd2c006b1b0 msgr2=0x7fd2c006d5c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:08.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:08.902+0000 7fd2dd6ca640 1 --2- 172.21.15.16:0/79553708 >> [v2:172.21.15.123:6800/2187102998,v1:172.21.15.123:6801/2187102998] conn(0x7fd2c006b1b0 0x7fd2c006d5c0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:08.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:08.902+0000 7fd2dd6ca640 1 -- 172.21.15.16:0/79553708 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fd2c0065430 msgr2=0x7fd2c00678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:08.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:08.902+0000 7fd2dd6ca640 1 --2- 172.21.15.16:0/79553708 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fd2c0065430 0x7fd2c00678e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd2c8009af0 tx=0x7fd2c800b040 comp rx=0 tx=0).stop 2024-06-18T02:58:08.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:08.902+0000 7fd2dd6ca640 1 -- 172.21.15.16:0/79553708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2d8102c40 msgr2=0x7fd2d8113560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:08.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:08.902+0000 7fd2dd6ca640 1 --2- 172.21.15.16:0/79553708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2d8102c40 0x7fd2d8113560 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc003080 tx=0x7fd2cc009da0 comp rx=0 tx=0).stop 2024-06-18T02:58:08.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:08.902+0000 7fd2dd6ca640 1 -- 172.21.15.16:0/79553708 shutdown_connections 2024-06-18T02:58:08.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:08.902+0000 7fd2dd6ca640 1 --2- 172.21.15.16:0/79553708 >> [v2:172.21.15.123:6800/2187102998,v1:172.21.15.123:6801/2187102998] conn(0x7fd2c006b1b0 0x7fd2c006d5c0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:08.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:08.902+0000 7fd2dd6ca640 1 --2- 172.21.15.16:0/79553708 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fd2c0065430 0x7fd2c00678e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:08.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:08.902+0000 7fd2dd6ca640 1 --2- 172.21.15.16:0/79553708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2d8102c40 0x7fd2d8113560 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:08.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:08.902+0000 7fd2dd6ca640 1 --2- 172.21.15.16:0/79553708 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2d81022a0 0x7fd2d8113020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:08.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:08.902+0000 7fd2dd6ca640 1 -- 172.21.15.16:0/79553708 >> 172.21.15.16:0/79553708 conn(0x7fd2d80f7e50 msgr2=0x7fd2d80f8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:58:08.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:08.902+0000 7fd2dd6ca640 1 -- 172.21.15.16:0/79553708 shutdown_connections 2024-06-18T02:58:08.911 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:08.902+0000 7fd2dd6ca640 1 -- 172.21.15.16:0/79553708 wait complete. 2024-06-18T02:58:08.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:58:08 smithi123 ceph-mon[36358]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:58:09.585 INFO:teuthology.orchestra.run.smithi016.stdout:115964117012 2024-06-18T02:58:09.586 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-06-18T02:58:09.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:09.850+0000 7f070814d640 1 -- 172.21.15.16:0/323472139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07000fb570 msgr2=0x7f07000fb940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:09.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:09.850+0000 7f070814d640 1 --2- 172.21.15.16:0/323472139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07000fb570 0x7f07000fb940 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f06fc0048d0 tx=0x7f06fc009da0 comp rx=0 tx=0).stop 2024-06-18T02:58:09.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:09.850+0000 7f070814d640 1 -- 172.21.15.16:0/323472139 shutdown_connections 2024-06-18T02:58:09.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:09.850+0000 7f070814d640 1 --2- 172.21.15.16:0/323472139 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f07000fbe80 0x7f07000fcd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:09.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:09.850+0000 7f070814d640 1 --2- 172.21.15.16:0/323472139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07000fb570 0x7f07000fb940 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:09.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:09.850+0000 7f070814d640 1 -- 172.21.15.16:0/323472139 >> 172.21.15.16:0/323472139 conn(0x7f0700072110 msgr2=0x7f0700072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:58:09.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:09.851+0000 7f070814d640 1 -- 172.21.15.16:0/323472139 shutdown_connections 2024-06-18T02:58:09.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:09.851+0000 7f070814d640 1 -- 172.21.15.16:0/323472139 wait complete. 2024-06-18T02:58:09.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:09.851+0000 7f070814d640 1 Processor -- start 2024-06-18T02:58:09.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:09.851+0000 7f070814d640 1 -- start start 2024-06-18T02:58:09.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:09.851+0000 7f070814d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f07000fb570 0x7f0700111f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:58:09.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:09.851+0000 7f070814d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07000fbe80 0x7f0700112460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:58:09.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:09.852+0000 7f070814d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0700112a60 con 0x7f07000fbe80 2024-06-18T02:58:09.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:09.852+0000 7f070814d640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f070010bea0 con 0x7f07000fb570 2024-06-18T02:58:09.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:09.852+0000 7f07056e8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07000fbe80 0x7f0700112460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:58:09.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:09.852+0000 7f07056e8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07000fbe80 0x7f0700112460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48614/0 (socket says 172.21.15.16:48614) 2024-06-18T02:58:09.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:09.852+0000 7f07056e8640 1 -- 172.21.15.16:0/2120301627 learned_addr learned my addr 172.21.15.16:0/2120301627 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:58:09.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:09.852+0000 7f0705ee9640 1 --2- 172.21.15.16:0/2120301627 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f07000fb570 0x7f0700111f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:58:09.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:09.852+0000 7f07056e8640 1 -- 172.21.15.16:0/2120301627 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f07000fb570 msgr2=0x7f0700111f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:09.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:09.852+0000 7f07056e8640 1 --2- 172.21.15.16:0/2120301627 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f07000fb570 0x7f0700111f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:09.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:09.852+0000 7f07056e8640 1 -- 172.21.15.16:0/2120301627 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06fc005040 con 0x7f07000fbe80 2024-06-18T02:58:09.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:09.852+0000 7f07056e8640 1 --2- 172.21.15.16:0/2120301627 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07000fbe80 0x7f0700112460 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f06f0002a50 tx=0x7f06f0004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:58:09.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:09.852+0000 7f06f67fc640 1 -- 172.21.15.16:0/2120301627 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f06f0007c30 con 0x7f07000fbe80 2024-06-18T02:58:09.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:09.852+0000 7f06f67fc640 1 -- 172.21.15.16:0/2120301627 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f06f000f040 con 0x7f07000fbe80 2024-06-18T02:58:09.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:09.852+0000 7f070814d640 1 -- 172.21.15.16:0/2120301627 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f070010c180 con 0x7f07000fbe80 2024-06-18T02:58:09.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:09.852+0000 7f06f67fc640 1 -- 172.21.15.16:0/2120301627 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f06f000db60 con 0x7f07000fbe80 2024-06-18T02:58:09.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:09.852+0000 7f070814d640 1 -- 172.21.15.16:0/2120301627 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f070010c720 con 0x7f07000fbe80 2024-06-18T02:58:09.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:09.853+0000 7f070814d640 1 -- 172.21.15.16:0/2120301627 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f070010ccc0 con 0x7f07000fbe80 2024-06-18T02:58:09.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:09.854+0000 7f06f67fc640 1 -- 172.21.15.16:0/2120301627 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f06f000dcc0 con 0x7f07000fbe80 2024-06-18T02:58:09.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:09.855+0000 7f06f67fc640 1 --2- 172.21.15.16:0/2120301627 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f06ec065430 0x7f06ec0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:58:09.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:09.855+0000 7f06f67fc640 1 -- 172.21.15.16:0/2120301627 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f06f0082320 con 0x7f07000fbe80 2024-06-18T02:58:09.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:09.855+0000 7f0705ee9640 1 --2- 172.21.15.16:0/2120301627 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f06ec065430 0x7f06ec0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:58:09.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:09.855+0000 7f06f67fc640 1 --2- 172.21.15.16:0/2120301627 >> [v2:172.21.15.123:6808/4038205871,v1:172.21.15.123:6809/4038205871] conn(0x7f06ec06b1b0 0x7f06ec06d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:58:09.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:09.855+0000 7f06f67fc640 1 -- 172.21.15.16:0/2120301627 --> [v2:172.21.15.123:6808/4038205871,v1:172.21.15.123:6809/4038205871] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f06ec06dc70 con 0x7f06ec06b1b0 2024-06-18T02:58:09.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:09.855+0000 7f06f67fc640 1 -- 172.21.15.16:0/2120301627 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f06f0084970 con 0x7f07000fbe80 2024-06-18T02:58:09.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:09.855+0000 7f07066ea640 1 --2- 172.21.15.16:0/2120301627 >> [v2:172.21.15.123:6808/4038205871,v1:172.21.15.123:6809/4038205871] conn(0x7f06ec06b1b0 0x7f06ec06d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:58:09.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:09.855+0000 7f0705ee9640 1 --2- 172.21.15.16:0/2120301627 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f06ec065430 0x7f06ec0678e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f06fc032430 tx=0x7f06fc035040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:58:09.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:09.856+0000 7f07066ea640 1 --2- 172.21.15.16:0/2120301627 >> [v2:172.21.15.123:6808/4038205871,v1:172.21.15.123:6809/4038205871] conn(0x7f06ec06b1b0 0x7f06ec06d5c0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:58:09.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:09.857+0000 7f06f67fc640 1 -- 172.21.15.16:0/2120301627 <== osd.4 v2:172.21.15.123:6808/4038205871 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f06ec06dc70 con 0x7f06ec06b1b0 2024-06-18T02:58:09.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:09.870+0000 7f070814d640 1 -- 172.21.15.16:0/2120301627 --> [v2:172.21.15.123:6808/4038205871,v1:172.21.15.123:6809/4038205871] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f07000f90c0 con 0x7f06ec06b1b0 2024-06-18T02:58:09.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:09.871+0000 7f06f67fc640 1 -- 172.21.15.16:0/2120301627 <== osd.4 v2:172.21.15.123:6808/4038205871 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f07000f90c0 con 0x7f06ec06b1b0 2024-06-18T02:58:09.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:09.871+0000 7f070814d640 1 -- 172.21.15.16:0/2120301627 >> [v2:172.21.15.123:6808/4038205871,v1:172.21.15.123:6809/4038205871] conn(0x7f06ec06b1b0 msgr2=0x7f06ec06d5c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:09.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:09.871+0000 7f070814d640 1 --2- 172.21.15.16:0/2120301627 >> [v2:172.21.15.123:6808/4038205871,v1:172.21.15.123:6809/4038205871] conn(0x7f06ec06b1b0 0x7f06ec06d5c0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:09.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:09.871+0000 7f070814d640 1 -- 172.21.15.16:0/2120301627 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f06ec065430 msgr2=0x7f06ec0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:09.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:09.871+0000 7f070814d640 1 --2- 172.21.15.16:0/2120301627 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f06ec065430 0x7f06ec0678e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f06fc032430 tx=0x7f06fc035040 comp rx=0 tx=0).stop 2024-06-18T02:58:09.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:09.871+0000 7f070814d640 1 -- 172.21.15.16:0/2120301627 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07000fbe80 msgr2=0x7f0700112460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:09.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:09.871+0000 7f070814d640 1 --2- 172.21.15.16:0/2120301627 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07000fbe80 0x7f0700112460 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f06f0002a50 tx=0x7f06f0004600 comp rx=0 tx=0).stop 2024-06-18T02:58:09.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:09.871+0000 7f070814d640 1 -- 172.21.15.16:0/2120301627 shutdown_connections 2024-06-18T02:58:09.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:09.871+0000 7f070814d640 1 --2- 172.21.15.16:0/2120301627 >> [v2:172.21.15.123:6808/4038205871,v1:172.21.15.123:6809/4038205871] conn(0x7f06ec06b1b0 0x7f06ec06d5c0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:09.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:09.871+0000 7f070814d640 1 --2- 172.21.15.16:0/2120301627 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f06ec065430 0x7f06ec0678e0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:09.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:09.871+0000 7f070814d640 1 --2- 172.21.15.16:0/2120301627 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07000fbe80 0x7f0700112460 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:09.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:09.871+0000 7f070814d640 1 --2- 172.21.15.16:0/2120301627 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f07000fb570 0x7f0700111f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:09.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:09.871+0000 7f070814d640 1 -- 172.21.15.16:0/2120301627 >> 172.21.15.16:0/2120301627 conn(0x7f0700072110 msgr2=0x7f07000f86b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:58:09.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:09.871+0000 7f070814d640 1 -- 172.21.15.16:0/2120301627 shutdown_connections 2024-06-18T02:58:09.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:09.871+0000 7f070814d640 1 -- 172.21.15.16:0/2120301627 wait complete. 2024-06-18T02:58:10.677 INFO:teuthology.orchestra.run.smithi016.stdout:150323855375 2024-06-18T02:58:10.678 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-06-18T02:58:10.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:58:10 smithi016 ceph-mon[29214]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:58:10.980 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:58:10 smithi123 ceph-mon[36358]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:58:11.060 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T02:58:11.610 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T02:58:11.638 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T02:58:11.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:58:11 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:58:11.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:58:11 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:58:12.927 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:58:12 smithi016 ceph-mon[29214]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:58:12.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:58:12 smithi123 ceph-mon[36358]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:58:13.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.220+0000 7fce507e9640 1 -- 172.21.15.16:0/364065778 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce480fdcc0 msgr2=0x7fce48106390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:13.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.220+0000 7fce507e9640 1 --2- 172.21.15.16:0/364065778 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce480fdcc0 0x7fce48106390 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fce3c0048d0 tx=0x7fce3c009da0 comp rx=0 tx=0).stop 2024-06-18T02:58:13.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.220+0000 7fce507e9640 1 -- 172.21.15.16:0/364065778 shutdown_connections 2024-06-18T02:58:13.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.220+0000 7fce507e9640 1 --2- 172.21.15.16:0/364065778 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce480fdcc0 0x7fce48106390 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:13.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.220+0000 7fce507e9640 1 --2- 172.21.15.16:0/364065778 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce480fd320 0x7fce480fd6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:13.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.220+0000 7fce507e9640 1 -- 172.21.15.16:0/364065778 >> 172.21.15.16:0/364065778 conn(0x7fce480f6dc0 msgr2=0x7fce480f91d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:58:13.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.220+0000 7fce507e9640 1 -- 172.21.15.16:0/364065778 shutdown_connections 2024-06-18T02:58:13.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.220+0000 7fce507e9640 1 -- 172.21.15.16:0/364065778 wait complete. 2024-06-18T02:58:13.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.221+0000 7fce507e9640 1 Processor -- start 2024-06-18T02:58:13.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.221+0000 7fce507e9640 1 -- start start 2024-06-18T02:58:13.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.221+0000 7fce507e9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce480fd320 0x7fce480fb1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:58:13.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.221+0000 7fce507e9640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce480fdcc0 0x7fce480fb720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:58:13.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.221+0000 7fce507e9640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce480fbc60 con 0x7fce480fd320 2024-06-18T02:58:13.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.221+0000 7fce507e9640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce480fbdd0 con 0x7fce480fdcc0 2024-06-18T02:58:13.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.221+0000 7fce4e585640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce480fd320 0x7fce480fb1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:58:13.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.221+0000 7fce4e585640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce480fd320 0x7fce480fb1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52006/0 (socket says 172.21.15.16:52006) 2024-06-18T02:58:13.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.221+0000 7fce4e585640 1 -- 172.21.15.16:0/3571278733 learned_addr learned my addr 172.21.15.16:0/3571278733 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:58:13.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.221+0000 7fce4dd84640 1 --2- 172.21.15.16:0/3571278733 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce480fdcc0 0x7fce480fb720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:58:13.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.222+0000 7fce4e585640 1 -- 172.21.15.16:0/3571278733 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce480fdcc0 msgr2=0x7fce480fb720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:13.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.222+0000 7fce4e585640 1 --2- 172.21.15.16:0/3571278733 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce480fdcc0 0x7fce480fb720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:13.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.222+0000 7fce4e585640 1 -- 172.21.15.16:0/3571278733 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce3c005040 con 0x7fce480fd320 2024-06-18T02:58:13.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.222+0000 7fce4dd84640 1 --2- 172.21.15.16:0/3571278733 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce480fdcc0 0x7fce480fb720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-18T02:58:13.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.222+0000 7fce4e585640 1 --2- 172.21.15.16:0/3571278733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce480fd320 0x7fce480fb1e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fce4400c790 tx=0x7fce440043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:58:13.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.222+0000 7fce3affd640 1 -- 172.21.15.16:0/3571278733 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce4400a820 con 0x7fce480fd320 2024-06-18T02:58:13.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.222+0000 7fce3affd640 1 -- 172.21.15.16:0/3571278733 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fce4400ae50 con 0x7fce480fd320 2024-06-18T02:58:13.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.222+0000 7fce507e9640 1 -- 172.21.15.16:0/3571278733 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce480fc060 con 0x7fce480fd320 2024-06-18T02:58:13.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.222+0000 7fce3affd640 1 -- 172.21.15.16:0/3571278733 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce440115e0 con 0x7fce480fd320 2024-06-18T02:58:13.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.222+0000 7fce507e9640 1 -- 172.21.15.16:0/3571278733 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce48119760 con 0x7fce480fd320 2024-06-18T02:58:13.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.223+0000 7fce507e9640 1 -- 172.21.15.16:0/3571278733 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fce48101660 con 0x7fce480fd320 2024-06-18T02:58:13.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.224+0000 7fce3affd640 1 -- 172.21.15.16:0/3571278733 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fce44012400 con 0x7fce480fd320 2024-06-18T02:58:13.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.224+0000 7fce3affd640 1 --2- 172.21.15.16:0/3571278733 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fce34065430 0x7fce340678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:58:13.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.224+0000 7fce3affd640 1 -- 172.21.15.16:0/3571278733 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fce44082690 con 0x7fce480fd320 2024-06-18T02:58:13.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.224+0000 7fce4dd84640 1 --2- 172.21.15.16:0/3571278733 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fce34065430 0x7fce340678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:58:13.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.224+0000 7fce3affd640 1 --2- 172.21.15.16:0/3571278733 >> [v2:172.21.15.16:6802/3634864038,v1:172.21.15.16:6803/3634864038] conn(0x7fce3406b1b0 0x7fce3406d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:58:13.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.224+0000 7fce3affd640 1 -- 172.21.15.16:0/3571278733 --> [v2:172.21.15.16:6802/3634864038,v1:172.21.15.16:6803/3634864038] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fce3406dc70 con 0x7fce3406b1b0 2024-06-18T02:58:13.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.224+0000 7fce3affd640 1 -- 172.21.15.16:0/3571278733 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7fce44084ce0 con 0x7fce480fd320 2024-06-18T02:58:13.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.224+0000 7fce4ed86640 1 --2- 172.21.15.16:0/3571278733 >> [v2:172.21.15.16:6802/3634864038,v1:172.21.15.16:6803/3634864038] conn(0x7fce3406b1b0 0x7fce3406d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:58:13.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.224+0000 7fce4dd84640 1 --2- 172.21.15.16:0/3571278733 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fce34065430 0x7fce340678e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fce3c007ac0 tx=0x7fce3c009da0 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:58:13.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.225+0000 7fce4ed86640 1 --2- 172.21.15.16:0/3571278733 >> [v2:172.21.15.16:6802/3634864038,v1:172.21.15.16:6803/3634864038] conn(0x7fce3406b1b0 0x7fce3406d5c0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:58:13.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.225+0000 7fce3affd640 1 -- 172.21.15.16:0/3571278733 <== osd.0 v2:172.21.15.16:6802/3634864038 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fce3406dc70 con 0x7fce3406b1b0 2024-06-18T02:58:13.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.239+0000 7fce507e9640 1 -- 172.21.15.16:0/3571278733 --> [v2:172.21.15.16:6802/3634864038,v1:172.21.15.16:6803/3634864038] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fce48065ff0 con 0x7fce3406b1b0 2024-06-18T02:58:13.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.239+0000 7fce3affd640 1 -- 172.21.15.16:0/3571278733 <== osd.0 v2:172.21.15.16:6802/3634864038 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fce48065ff0 con 0x7fce3406b1b0 2024-06-18T02:58:13.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.239+0000 7fce507e9640 1 -- 172.21.15.16:0/3571278733 >> [v2:172.21.15.16:6802/3634864038,v1:172.21.15.16:6803/3634864038] conn(0x7fce3406b1b0 msgr2=0x7fce3406d5c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:13.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.239+0000 7fce507e9640 1 --2- 172.21.15.16:0/3571278733 >> [v2:172.21.15.16:6802/3634864038,v1:172.21.15.16:6803/3634864038] conn(0x7fce3406b1b0 0x7fce3406d5c0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:13.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.239+0000 7fce507e9640 1 -- 172.21.15.16:0/3571278733 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fce34065430 msgr2=0x7fce340678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:13.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.239+0000 7fce507e9640 1 --2- 172.21.15.16:0/3571278733 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fce34065430 0x7fce340678e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fce3c007ac0 tx=0x7fce3c009da0 comp rx=0 tx=0).stop 2024-06-18T02:58:13.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.239+0000 7fce507e9640 1 -- 172.21.15.16:0/3571278733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce480fd320 msgr2=0x7fce480fb1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:13.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.239+0000 7fce507e9640 1 --2- 172.21.15.16:0/3571278733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce480fd320 0x7fce480fb1e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fce4400c790 tx=0x7fce440043c0 comp rx=0 tx=0).stop 2024-06-18T02:58:13.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.239+0000 7fce507e9640 1 -- 172.21.15.16:0/3571278733 shutdown_connections 2024-06-18T02:58:13.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.240+0000 7fce507e9640 1 --2- 172.21.15.16:0/3571278733 >> [v2:172.21.15.16:6802/3634864038,v1:172.21.15.16:6803/3634864038] conn(0x7fce3406b1b0 0x7fce3406d5c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:13.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.240+0000 7fce507e9640 1 --2- 172.21.15.16:0/3571278733 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fce34065430 0x7fce340678e0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:13.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.240+0000 7fce507e9640 1 --2- 172.21.15.16:0/3571278733 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce480fdcc0 0x7fce480fb720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:13.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.240+0000 7fce507e9640 1 --2- 172.21.15.16:0/3571278733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fce480fd320 0x7fce480fb1e0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:13.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.240+0000 7fce507e9640 1 -- 172.21.15.16:0/3571278733 >> 172.21.15.16:0/3571278733 conn(0x7fce480f6dc0 msgr2=0x7fce48100c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:58:13.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.240+0000 7fce507e9640 1 -- 172.21.15.16:0/3571278733 shutdown_connections 2024-06-18T02:58:13.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.240+0000 7fce507e9640 1 -- 172.21.15.16:0/3571278733 wait complete. 2024-06-18T02:58:14.721 INFO:teuthology.orchestra.run.smithi016.stdout:38654705699 2024-06-18T02:58:14.721 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-06-18T02:58:14.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.747+0000 7f1660af7640 1 -- 172.21.15.16:0/2403546443 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f165c1022a0 msgr2=0x7f165c102670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:14.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.747+0000 7f1660af7640 1 --2- 172.21.15.16:0/2403546443 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f165c1022a0 0x7f165c102670 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f164c0048d0 tx=0x7f164c009da0 comp rx=0 tx=0).stop 2024-06-18T02:58:14.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.747+0000 7f1660af7640 1 -- 172.21.15.16:0/2403546443 shutdown_connections 2024-06-18T02:58:14.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.748+0000 7f1660af7640 1 --2- 172.21.15.16:0/2403546443 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f165c102c40 0x7f165c0fc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:14.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.748+0000 7f1660af7640 1 --2- 172.21.15.16:0/2403546443 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f165c1022a0 0x7f165c102670 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:14.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.748+0000 7f1660af7640 1 -- 172.21.15.16:0/2403546443 >> 172.21.15.16:0/2403546443 conn(0x7f165c0f7e70 msgr2=0x7f165c0fa280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:58:14.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.748+0000 7f1660af7640 1 -- 172.21.15.16:0/2403546443 shutdown_connections 2024-06-18T02:58:14.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.748+0000 7f1660af7640 1 -- 172.21.15.16:0/2403546443 wait complete. 2024-06-18T02:58:14.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.748+0000 7f1660af7640 1 Processor -- start 2024-06-18T02:58:14.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.748+0000 7f1660af7640 1 -- start start 2024-06-18T02:58:14.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.748+0000 7f1660af7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f165c1022a0 0x7f165c112fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:58:14.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.748+0000 7f1660af7640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f165c102c40 0x7f165c113520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:58:14.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.748+0000 7f1660af7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f165c113b20 con 0x7f165c1022a0 2024-06-18T02:58:14.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.748+0000 7f1660af7640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f165c10cf60 con 0x7f165c102c40 2024-06-18T02:58:14.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.748+0000 7f165a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f165c1022a0 0x7f165c112fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:58:14.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.748+0000 7f165a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f165c1022a0 0x7f165c112fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52022/0 (socket says 172.21.15.16:52022) 2024-06-18T02:58:14.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.748+0000 7f165a59c640 1 -- 172.21.15.16:0/2447403870 learned_addr learned my addr 172.21.15.16:0/2447403870 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:58:14.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.749+0000 7f1659d9b640 1 --2- 172.21.15.16:0/2447403870 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f165c102c40 0x7f165c113520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:58:14.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.749+0000 7f165a59c640 1 -- 172.21.15.16:0/2447403870 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f165c102c40 msgr2=0x7f165c113520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:14.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.749+0000 7f165a59c640 1 --2- 172.21.15.16:0/2447403870 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f165c102c40 0x7f165c113520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:14.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.749+0000 7f165a59c640 1 -- 172.21.15.16:0/2447403870 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f164c005040 con 0x7f165c1022a0 2024-06-18T02:58:14.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.749+0000 7f1659d9b640 1 --2- 172.21.15.16:0/2447403870 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f165c102c40 0x7f165c113520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-18T02:58:14.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.749+0000 7f165a59c640 1 --2- 172.21.15.16:0/2447403870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f165c1022a0 0x7f165c112fe0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f164c0043d0 tx=0x7f164c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:58:14.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.749+0000 7f164affd640 1 -- 172.21.15.16:0/2447403870 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f164c038070 con 0x7f165c1022a0 2024-06-18T02:58:14.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.749+0000 7f164affd640 1 -- 172.21.15.16:0/2447403870 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f164c032970 con 0x7f165c1022a0 2024-06-18T02:58:14.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.749+0000 7f1660af7640 1 -- 172.21.15.16:0/2447403870 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f165c10d1e0 con 0x7f165c1022a0 2024-06-18T02:58:14.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.749+0000 7f164affd640 1 -- 172.21.15.16:0/2447403870 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f164c03da60 con 0x7f165c1022a0 2024-06-18T02:58:14.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.749+0000 7f1660af7640 1 -- 172.21.15.16:0/2447403870 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f165c10d6a0 con 0x7f165c1022a0 2024-06-18T02:58:14.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.750+0000 7f1660af7640 1 -- 172.21.15.16:0/2447403870 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f165c10dc40 con 0x7f165c1022a0 2024-06-18T02:58:14.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.751+0000 7f164affd640 1 -- 172.21.15.16:0/2447403870 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f164c030020 con 0x7f165c1022a0 2024-06-18T02:58:14.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.751+0000 7f164affd640 1 --2- 172.21.15.16:0/2447403870 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f1644065430 0x7f16440678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:58:14.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.751+0000 7f164affd640 1 -- 172.21.15.16:0/2447403870 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f164c0a26b0 con 0x7f165c1022a0 2024-06-18T02:58:14.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.751+0000 7f1659d9b640 1 --2- 172.21.15.16:0/2447403870 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f1644065430 0x7f16440678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:58:14.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.751+0000 7f164affd640 1 --2- 172.21.15.16:0/2447403870 >> [v2:172.21.15.16:6810/500143512,v1:172.21.15.16:6811/500143512] conn(0x7f164406b1b0 0x7f164406d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:58:14.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.751+0000 7f164affd640 1 -- 172.21.15.16:0/2447403870 --> [v2:172.21.15.16:6810/500143512,v1:172.21.15.16:6811/500143512] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f164406dc70 con 0x7f164406b1b0 2024-06-18T02:58:14.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.751+0000 7f164affd640 1 -- 172.21.15.16:0/2447403870 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f164c0a4ca0 con 0x7f165c1022a0 2024-06-18T02:58:14.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.751+0000 7f165ad9d640 1 --2- 172.21.15.16:0/2447403870 >> [v2:172.21.15.16:6810/500143512,v1:172.21.15.16:6811/500143512] conn(0x7f164406b1b0 0x7f164406d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:58:14.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.752+0000 7f1659d9b640 1 --2- 172.21.15.16:0/2447403870 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f1644065430 0x7f16440678e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f165c10e8a0 tx=0x7f1650009ed0 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:58:14.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.752+0000 7f165ad9d640 1 --2- 172.21.15.16:0/2447403870 >> [v2:172.21.15.16:6810/500143512,v1:172.21.15.16:6811/500143512] conn(0x7f164406b1b0 0x7f164406d5c0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:58:14.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.752+0000 7f164affd640 1 -- 172.21.15.16:0/2447403870 <== osd.1 v2:172.21.15.16:6810/500143512 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f164406dc70 con 0x7f164406b1b0 2024-06-18T02:58:14.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.765+0000 7f1660af7640 1 -- 172.21.15.16:0/2447403870 --> [v2:172.21.15.16:6810/500143512,v1:172.21.15.16:6811/500143512] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f165c10de00 con 0x7f164406b1b0 2024-06-18T02:58:14.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.766+0000 7f164affd640 1 -- 172.21.15.16:0/2447403870 <== osd.1 v2:172.21.15.16:6810/500143512 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f165c10de00 con 0x7f164406b1b0 2024-06-18T02:58:14.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.766+0000 7f1660af7640 1 -- 172.21.15.16:0/2447403870 >> [v2:172.21.15.16:6810/500143512,v1:172.21.15.16:6811/500143512] conn(0x7f164406b1b0 msgr2=0x7f164406d5c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:14.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.766+0000 7f1660af7640 1 --2- 172.21.15.16:0/2447403870 >> [v2:172.21.15.16:6810/500143512,v1:172.21.15.16:6811/500143512] conn(0x7f164406b1b0 0x7f164406d5c0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:14.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.766+0000 7f1660af7640 1 -- 172.21.15.16:0/2447403870 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f1644065430 msgr2=0x7f16440678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:14.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.766+0000 7f1660af7640 1 --2- 172.21.15.16:0/2447403870 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f1644065430 0x7f16440678e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f165c10e8a0 tx=0x7f1650009ed0 comp rx=0 tx=0).stop 2024-06-18T02:58:14.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.766+0000 7f1660af7640 1 -- 172.21.15.16:0/2447403870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f165c1022a0 msgr2=0x7f165c112fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:14.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.766+0000 7f1660af7640 1 --2- 172.21.15.16:0/2447403870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f165c1022a0 0x7f165c112fe0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f164c0043d0 tx=0x7f164c009da0 comp rx=0 tx=0).stop 2024-06-18T02:58:14.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.766+0000 7f1660af7640 1 -- 172.21.15.16:0/2447403870 shutdown_connections 2024-06-18T02:58:14.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.766+0000 7f1660af7640 1 --2- 172.21.15.16:0/2447403870 >> [v2:172.21.15.16:6810/500143512,v1:172.21.15.16:6811/500143512] conn(0x7f164406b1b0 0x7f164406d5c0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:14.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.766+0000 7f1660af7640 1 --2- 172.21.15.16:0/2447403870 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f1644065430 0x7f16440678e0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:14.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.766+0000 7f1660af7640 1 --2- 172.21.15.16:0/2447403870 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f165c102c40 0x7f165c113520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:14.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.766+0000 7f1660af7640 1 --2- 172.21.15.16:0/2447403870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f165c1022a0 0x7f165c112fe0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:14.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.766+0000 7f1660af7640 1 -- 172.21.15.16:0/2447403870 >> 172.21.15.16:0/2447403870 conn(0x7f165c0f7e70 msgr2=0x7f165c0f8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:58:14.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.766+0000 7f1660af7640 1 -- 172.21.15.16:0/2447403870 shutdown_connections 2024-06-18T02:58:14.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:13.766+0000 7f1660af7640 1 -- 172.21.15.16:0/2447403870 wait complete. 2024-06-18T02:58:14.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:58:14 smithi123 ceph-mon[36358]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:58:15.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:58:14 smithi016 ceph-mon[29214]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:58:16.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:58:16 smithi016 ceph-mon[29214]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:58:16.868 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T02:58:16.902 INFO:teuthology.orchestra.run.smithi016.stdout:60129542175 2024-06-18T02:58:16.903 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-06-18T02:58:16.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:58:16 smithi123 ceph-mon[36358]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:58:17.314 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T02:58:18.894 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:58:18 smithi016 ceph-mon[29214]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:58:18.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:58:18 smithi123 ceph-mon[36358]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:58:19.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.491+0000 7fdb7ad4d640 1 -- 172.21.15.16:0/2119093865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb741022f0 msgr2=0x7fdb741026c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:19.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.491+0000 7fdb7ad4d640 1 --2- 172.21.15.16:0/2119093865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb741022f0 0x7fdb741026c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fdb680048d0 tx=0x7fdb68009da0 comp rx=0 tx=0).stop 2024-06-18T02:58:19.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.491+0000 7fdb7ad4d640 1 -- 172.21.15.16:0/2119093865 shutdown_connections 2024-06-18T02:58:19.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.491+0000 7fdb7ad4d640 1 --2- 172.21.15.16:0/2119093865 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb74102c90 0x7fdb740fc320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:19.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.491+0000 7fdb7ad4d640 1 --2- 172.21.15.16:0/2119093865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb741022f0 0x7fdb741026c0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:19.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.491+0000 7fdb7ad4d640 1 -- 172.21.15.16:0/2119093865 >> 172.21.15.16:0/2119093865 conn(0x7fdb740f7f00 msgr2=0x7fdb740fa310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:58:19.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.492+0000 7fdb7ad4d640 1 -- 172.21.15.16:0/2119093865 shutdown_connections 2024-06-18T02:58:19.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.492+0000 7fdb7ad4d640 1 -- 172.21.15.16:0/2119093865 wait complete. 2024-06-18T02:58:19.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.492+0000 7fdb7ad4d640 1 Processor -- start 2024-06-18T02:58:19.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.492+0000 7fdb7ad4d640 1 -- start start 2024-06-18T02:58:19.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.493+0000 7fdb7ad4d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb741022f0 0x7fdb74112ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:58:19.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.493+0000 7fdb7ad4d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb74102c90 0x7fdb74113530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:58:19.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.493+0000 7fdb7ad4d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb74113b30 con 0x7fdb74102c90 2024-06-18T02:58:19.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.493+0000 7fdb7ad4d640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb7410cf70 con 0x7fdb741022f0 2024-06-18T02:58:19.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.493+0000 7fdb73fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb74102c90 0x7fdb74113530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:58:19.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.493+0000 7fdb73fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb74102c90 0x7fdb74113530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52040/0 (socket says 172.21.15.16:52040) 2024-06-18T02:58:19.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.493+0000 7fdb73fff640 1 -- 172.21.15.16:0/624409682 learned_addr learned my addr 172.21.15.16:0/624409682 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:58:19.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.493+0000 7fdb78ae9640 1 --2- 172.21.15.16:0/624409682 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb741022f0 0x7fdb74112ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:58:19.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.493+0000 7fdb73fff640 1 -- 172.21.15.16:0/624409682 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb741022f0 msgr2=0x7fdb74112ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:19.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.493+0000 7fdb73fff640 1 --2- 172.21.15.16:0/624409682 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb741022f0 0x7fdb74112ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:19.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.493+0000 7fdb73fff640 1 -- 172.21.15.16:0/624409682 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb68005040 con 0x7fdb74102c90 2024-06-18T02:58:19.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.493+0000 7fdb73fff640 1 --2- 172.21.15.16:0/624409682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb74102c90 0x7fdb74113530 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fdb6000c790 tx=0x7fdb600043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:58:19.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.493+0000 7fdb717fa640 1 -- 172.21.15.16:0/624409682 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb6000a820 con 0x7fdb74102c90 2024-06-18T02:58:19.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.493+0000 7fdb7ad4d640 1 -- 172.21.15.16:0/624409682 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb7410d250 con 0x7fdb74102c90 2024-06-18T02:58:19.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.493+0000 7fdb717fa640 1 -- 172.21.15.16:0/624409682 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdb6000ae50 con 0x7fdb74102c90 2024-06-18T02:58:19.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.493+0000 7fdb717fa640 1 -- 172.21.15.16:0/624409682 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb600115e0 con 0x7fdb74102c90 2024-06-18T02:58:19.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.494+0000 7fdb7ad4d640 1 -- 172.21.15.16:0/624409682 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb7410d7f0 con 0x7fdb74102c90 2024-06-18T02:58:19.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.495+0000 7fdb7ad4d640 1 -- 172.21.15.16:0/624409682 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb740fda60 con 0x7fdb74102c90 2024-06-18T02:58:19.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.495+0000 7fdb717fa640 1 -- 172.21.15.16:0/624409682 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fdb60012400 con 0x7fdb74102c90 2024-06-18T02:58:19.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.496+0000 7fdb717fa640 1 --2- 172.21.15.16:0/624409682 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fdb64065430 0x7fdb640678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:58:19.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.496+0000 7fdb717fa640 1 -- 172.21.15.16:0/624409682 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fdb60082690 con 0x7fdb74102c90 2024-06-18T02:58:19.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.496+0000 7fdb78ae9640 1 --2- 172.21.15.16:0/624409682 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fdb64065430 0x7fdb640678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:58:19.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.496+0000 7fdb78ae9640 1 --2- 172.21.15.16:0/624409682 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fdb64065430 0x7fdb640678e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fdb68000c00 tx=0x7fdb68035040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:58:19.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.500+0000 7fdb717fa640 1 -- 172.21.15.16:0/624409682 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fdb60050bc0 con 0x7fdb74102c90 2024-06-18T02:58:19.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.680+0000 7fdb7ad4d640 1 -- 172.21.15.16:0/624409682 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fdb74109450 con 0x7fdb74102c90 2024-06-18T02:58:19.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.680+0000 7fdb717fa640 1 -- 172.21.15.16:0/624409682 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fdb600541e0 con 0x7fdb74102c90 2024-06-18T02:58:19.681 INFO:teuthology.orchestra.run.smithi016.stdout:81604378650 2024-06-18T02:58:19.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.682+0000 7fdb7ad4d640 1 -- 172.21.15.16:0/624409682 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fdb64065430 msgr2=0x7fdb640678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:19.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.683+0000 7fdb7ad4d640 1 --2- 172.21.15.16:0/624409682 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fdb64065430 0x7fdb640678e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fdb68000c00 tx=0x7fdb68035040 comp rx=0 tx=0).stop 2024-06-18T02:58:19.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.683+0000 7fdb7ad4d640 1 -- 172.21.15.16:0/624409682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb74102c90 msgr2=0x7fdb74113530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:19.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.683+0000 7fdb7ad4d640 1 --2- 172.21.15.16:0/624409682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb74102c90 0x7fdb74113530 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fdb6000c790 tx=0x7fdb600043c0 comp rx=0 tx=0).stop 2024-06-18T02:58:19.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.683+0000 7fdb7ad4d640 1 -- 172.21.15.16:0/624409682 shutdown_connections 2024-06-18T02:58:19.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.683+0000 7fdb7ad4d640 1 --2- 172.21.15.16:0/624409682 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fdb64065430 0x7fdb640678e0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:19.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.683+0000 7fdb7ad4d640 1 --2- 172.21.15.16:0/624409682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb74102c90 0x7fdb74113530 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:19.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.683+0000 7fdb7ad4d640 1 --2- 172.21.15.16:0/624409682 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb741022f0 0x7fdb74112ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:19.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.683+0000 7fdb7ad4d640 1 -- 172.21.15.16:0/624409682 >> 172.21.15.16:0/624409682 conn(0x7fdb740f7f00 msgr2=0x7fdb740f8e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:58:19.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.683+0000 7fdb7ad4d640 1 -- 172.21.15.16:0/624409682 shutdown_connections 2024-06-18T02:58:19.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.683+0000 7fdb7ad4d640 1 -- 172.21.15.16:0/624409682 wait complete. 2024-06-18T02:58:19.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.738+0000 7f5ca196a640 1 -- 172.21.15.16:0/609728929 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c9c0fdc70 msgr2=0x7f5c9c106340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:19.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.738+0000 7f5ca196a640 1 --2- 172.21.15.16:0/609728929 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c9c0fdc70 0x7f5c9c106340 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f5c900048d0 tx=0x7f5c90009da0 comp rx=0 tx=0).stop 2024-06-18T02:58:19.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.738+0000 7f5ca196a640 1 -- 172.21.15.16:0/609728929 shutdown_connections 2024-06-18T02:58:19.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.738+0000 7f5ca196a640 1 --2- 172.21.15.16:0/609728929 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c9c0fdc70 0x7f5c9c106340 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:19.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.738+0000 7f5ca196a640 1 --2- 172.21.15.16:0/609728929 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c9c0fd2d0 0x7f5c9c0fd6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:19.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.738+0000 7f5ca196a640 1 -- 172.21.15.16:0/609728929 >> 172.21.15.16:0/609728929 conn(0x7f5c9c0f6d50 msgr2=0x7f5c9c0f9160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:58:19.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.738+0000 7f5ca196a640 1 -- 172.21.15.16:0/609728929 shutdown_connections 2024-06-18T02:58:19.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.738+0000 7f5ca196a640 1 -- 172.21.15.16:0/609728929 wait complete. 2024-06-18T02:58:19.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.738+0000 7f5ca196a640 1 Processor -- start 2024-06-18T02:58:19.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.738+0000 7f5ca196a640 1 -- start start 2024-06-18T02:58:19.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.739+0000 7f5ca196a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c9c0fd2d0 0x7f5c9c113130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:58:19.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.739+0000 7f5ca196a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c9c0fdc70 0x7f5c9c113670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:58:19.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.739+0000 7f5ca196a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c9c113c70 con 0x7f5c9c0fdc70 2024-06-18T02:58:19.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.739+0000 7f5ca196a640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c9c10d0b0 con 0x7f5c9c0fd2d0 2024-06-18T02:58:19.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.739+0000 7f5c9a7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c9c0fdc70 0x7f5c9c113670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:58:19.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.739+0000 7f5c9a7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c9c0fdc70 0x7f5c9c113670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52056/0 (socket says 172.21.15.16:52056) 2024-06-18T02:58:19.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.739+0000 7f5c9affd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c9c0fd2d0 0x7f5c9c113130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:58:19.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.739+0000 7f5c9a7fc640 1 -- 172.21.15.16:0/1883358733 learned_addr learned my addr 172.21.15.16:0/1883358733 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:58:19.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.739+0000 7f5c9a7fc640 1 -- 172.21.15.16:0/1883358733 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c9c0fd2d0 msgr2=0x7f5c9c113130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:19.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.739+0000 7f5c9a7fc640 1 --2- 172.21.15.16:0/1883358733 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c9c0fd2d0 0x7f5c9c113130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:19.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.739+0000 7f5c9a7fc640 1 -- 172.21.15.16:0/1883358733 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c90005040 con 0x7f5c9c0fdc70 2024-06-18T02:58:19.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.739+0000 7f5c9a7fc640 1 --2- 172.21.15.16:0/1883358733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c9c0fdc70 0x7f5c9c113670 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f5c90000c00 tx=0x7f5c90009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:58:19.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.740+0000 7f5c837fe640 1 -- 172.21.15.16:0/1883358733 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c90038070 con 0x7f5c9c0fdc70 2024-06-18T02:58:19.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.740+0000 7f5ca196a640 1 -- 172.21.15.16:0/1883358733 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c9c10d330 con 0x7f5c9c0fdc70 2024-06-18T02:58:19.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.740+0000 7f5c837fe640 1 -- 172.21.15.16:0/1883358733 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5c9003d940 con 0x7f5c9c0fdc70 2024-06-18T02:58:19.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.740+0000 7f5c837fe640 1 -- 172.21.15.16:0/1883358733 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c900339e0 con 0x7f5c9c0fdc70 2024-06-18T02:58:19.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.740+0000 7f5ca196a640 1 -- 172.21.15.16:0/1883358733 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c9c10d7f0 con 0x7f5c9c0fdc70 2024-06-18T02:58:19.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.741+0000 7f5ca196a640 1 -- 172.21.15.16:0/1883358733 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c9c066070 con 0x7f5c9c0fdc70 2024-06-18T02:58:19.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.741+0000 7f5c837fe640 1 -- 172.21.15.16:0/1883358733 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f5c90030020 con 0x7f5c9c0fdc70 2024-06-18T02:58:19.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.741+0000 7f5c837fe640 1 --2- 172.21.15.16:0/1883358733 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f5c84065430 0x7f5c840678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:58:19.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.742+0000 7f5c837fe640 1 -- 172.21.15.16:0/1883358733 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f5c900a2530 con 0x7f5c9c0fdc70 2024-06-18T02:58:19.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.742+0000 7f5c9affd640 1 --2- 172.21.15.16:0/1883358733 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f5c84065430 0x7f5c840678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:58:19.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.742+0000 7f5c9affd640 1 --2- 172.21.15.16:0/1883358733 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f5c84065430 0x7f5c840678e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c009af0 tx=0x7f5c8c00b040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:58:19.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.746+0000 7f5c837fe640 1 -- 172.21.15.16:0/1883358733 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f5c90070940 con 0x7f5c9c0fdc70 2024-06-18T02:58:19.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.921+0000 7f5ca196a640 1 -- 172.21.15.16:0/1883358733 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f5c9c0fa990 con 0x7f5c9c0fdc70 2024-06-18T02:58:19.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.921+0000 7f5c837fe640 1 -- 172.21.15.16:0/1883358733 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f5c90073f60 con 0x7f5c9c0fdc70 2024-06-18T02:58:19.922 INFO:teuthology.orchestra.run.smithi016.stdout:115964117014 2024-06-18T02:58:19.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.923+0000 7f5ca196a640 1 -- 172.21.15.16:0/1883358733 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f5c84065430 msgr2=0x7f5c840678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:19.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.923+0000 7f5ca196a640 1 --2- 172.21.15.16:0/1883358733 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f5c84065430 0x7f5c840678e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c009af0 tx=0x7f5c8c00b040 comp rx=0 tx=0).stop 2024-06-18T02:58:19.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.923+0000 7f5ca196a640 1 -- 172.21.15.16:0/1883358733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c9c0fdc70 msgr2=0x7f5c9c113670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:19.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.923+0000 7f5ca196a640 1 --2- 172.21.15.16:0/1883358733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c9c0fdc70 0x7f5c9c113670 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f5c90000c00 tx=0x7f5c90009da0 comp rx=0 tx=0).stop 2024-06-18T02:58:19.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.923+0000 7f5ca196a640 1 -- 172.21.15.16:0/1883358733 shutdown_connections 2024-06-18T02:58:19.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.923+0000 7f5ca196a640 1 --2- 172.21.15.16:0/1883358733 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f5c84065430 0x7f5c840678e0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:19.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.923+0000 7f5ca196a640 1 --2- 172.21.15.16:0/1883358733 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c9c0fdc70 0x7f5c9c113670 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:19.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.923+0000 7f5ca196a640 1 --2- 172.21.15.16:0/1883358733 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c9c0fd2d0 0x7f5c9c113130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:19.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.923+0000 7f5ca196a640 1 -- 172.21.15.16:0/1883358733 >> 172.21.15.16:0/1883358733 conn(0x7f5c9c0f6d50 msgr2=0x7f5c9c102570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:58:19.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.923+0000 7f5ca196a640 1 -- 172.21.15.16:0/1883358733 shutdown_connections 2024-06-18T02:58:19.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:19.923+0000 7f5ca196a640 1 -- 172.21.15.16:0/1883358733 wait complete. 2024-06-18T02:58:20.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:58:20 smithi123 ceph-mon[36358]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:58:20.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:58:20 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/624409682' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-06-18T02:58:20.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:58:20 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/1883358733' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-06-18T02:58:20.759 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:58:20 smithi016 ceph-mon[29214]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:58:20.759 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:58:20 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/624409682' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-06-18T02:58:20.759 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:58:20 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/1883358733' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-06-18T02:58:20.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:20.758+0000 7f9def3a8640 1 -- 172.21.15.16:0/1363372212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9de81022a0 msgr2=0x7f9de8102670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:20.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:20.758+0000 7f9def3a8640 1 --2- 172.21.15.16:0/1363372212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9de81022a0 0x7f9de8102670 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f9de40048d0 tx=0x7f9de4009da0 comp rx=0 tx=0).stop 2024-06-18T02:58:20.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:20.759+0000 7f9def3a8640 1 -- 172.21.15.16:0/1363372212 shutdown_connections 2024-06-18T02:58:20.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:20.759+0000 7f9def3a8640 1 --2- 172.21.15.16:0/1363372212 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9de8102c40 0x7f9de80fc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:20.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:20.759+0000 7f9def3a8640 1 --2- 172.21.15.16:0/1363372212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9de81022a0 0x7f9de8102670 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:20.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:20.759+0000 7f9def3a8640 1 -- 172.21.15.16:0/1363372212 >> 172.21.15.16:0/1363372212 conn(0x7f9de80f7e50 msgr2=0x7f9de80fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:58:20.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:20.759+0000 7f9def3a8640 1 -- 172.21.15.16:0/1363372212 shutdown_connections 2024-06-18T02:58:20.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:20.759+0000 7f9def3a8640 1 -- 172.21.15.16:0/1363372212 wait complete. 2024-06-18T02:58:20.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:20.759+0000 7f9def3a8640 1 Processor -- start 2024-06-18T02:58:20.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:20.759+0000 7f9def3a8640 1 -- start start 2024-06-18T02:58:20.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:20.760+0000 7f9def3a8640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9de81022a0 0x7f9de806f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:58:20.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:20.760+0000 7f9def3a8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9de8102c40 0x7f9de806fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:58:20.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:20.760+0000 7f9def3a8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9de80701f0 con 0x7f9de8102c40 2024-06-18T02:58:20.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:20.760+0000 7f9def3a8640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9de8070360 con 0x7f9de81022a0 2024-06-18T02:58:20.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:20.760+0000 7f9dec943640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9de8102c40 0x7f9de806fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:58:20.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:20.760+0000 7f9dec943640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9de8102c40 0x7f9de806fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52078/0 (socket says 172.21.15.16:52078) 2024-06-18T02:58:20.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:20.760+0000 7f9dec943640 1 -- 172.21.15.16:0/4217576131 learned_addr learned my addr 172.21.15.16:0/4217576131 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:58:20.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:20.760+0000 7f9ded144640 1 --2- 172.21.15.16:0/4217576131 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9de81022a0 0x7f9de806f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:58:20.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:20.760+0000 7f9dec943640 1 -- 172.21.15.16:0/4217576131 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9de81022a0 msgr2=0x7f9de806f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:20.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:20.760+0000 7f9dec943640 1 --2- 172.21.15.16:0/4217576131 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9de81022a0 0x7f9de806f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:20.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:20.760+0000 7f9dec943640 1 -- 172.21.15.16:0/4217576131 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9de4005040 con 0x7f9de8102c40 2024-06-18T02:58:20.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:20.760+0000 7f9dec943640 1 --2- 172.21.15.16:0/4217576131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9de8102c40 0x7f9de806fcb0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f9dd8002a50 tx=0x7f9dd8004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:58:20.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:20.760+0000 7f9dddffb640 1 -- 172.21.15.16:0/4217576131 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9dd8007c30 con 0x7f9de8102c40 2024-06-18T02:58:20.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:20.760+0000 7f9def3a8640 1 -- 172.21.15.16:0/4217576131 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9de80705f0 con 0x7f9de8102c40 2024-06-18T02:58:20.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:20.760+0000 7f9dddffb640 1 -- 172.21.15.16:0/4217576131 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9dd800f040 con 0x7f9de8102c40 2024-06-18T02:58:20.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:20.760+0000 7f9dddffb640 1 -- 172.21.15.16:0/4217576131 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9dd800db60 con 0x7f9de8102c40 2024-06-18T02:58:20.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:20.760+0000 7f9def3a8640 1 -- 172.21.15.16:0/4217576131 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9de81196e0 con 0x7f9de8102c40 2024-06-18T02:58:20.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:20.761+0000 7f9def3a8640 1 -- 172.21.15.16:0/4217576131 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9de80fda10 con 0x7f9de8102c40 2024-06-18T02:58:20.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:20.762+0000 7f9dddffb640 1 -- 172.21.15.16:0/4217576131 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f9dd800dcc0 con 0x7f9de8102c40 2024-06-18T02:58:20.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:20.762+0000 7f9dddffb640 1 --2- 172.21.15.16:0/4217576131 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f9dd4065430 0x7f9dd40678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:58:20.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:20.762+0000 7f9dddffb640 1 -- 172.21.15.16:0/4217576131 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f9dd8082310 con 0x7f9de8102c40 2024-06-18T02:58:20.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:20.762+0000 7f9ded144640 1 --2- 172.21.15.16:0/4217576131 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f9dd4065430 0x7f9dd40678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:58:20.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:20.763+0000 7f9ded144640 1 --2- 172.21.15.16:0/4217576131 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f9dd4065430 0x7f9dd40678e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9de4009f80 tx=0x7f9de4035040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:58:20.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:20.767+0000 7f9dddffb640 1 -- 172.21.15.16:0/4217576131 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f9dd80507c0 con 0x7f9de8102c40 2024-06-18T02:58:20.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:20.941+0000 7f9def3a8640 1 -- 172.21.15.16:0/4217576131 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f9de80fba90 con 0x7f9de8102c40 2024-06-18T02:58:20.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:20.941+0000 7f9dddffb640 1 -- 172.21.15.16:0/4217576131 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f9dd8053de0 con 0x7f9de8102c40 2024-06-18T02:58:20.942 INFO:teuthology.orchestra.run.smithi016.stdout:171798691852 2024-06-18T02:58:20.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:20.943+0000 7f9def3a8640 1 -- 172.21.15.16:0/4217576131 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f9dd4065430 msgr2=0x7f9dd40678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:20.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:20.943+0000 7f9def3a8640 1 --2- 172.21.15.16:0/4217576131 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f9dd4065430 0x7f9dd40678e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9de4009f80 tx=0x7f9de4035040 comp rx=0 tx=0).stop 2024-06-18T02:58:20.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:20.943+0000 7f9def3a8640 1 -- 172.21.15.16:0/4217576131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9de8102c40 msgr2=0x7f9de806fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:20.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:20.943+0000 7f9def3a8640 1 --2- 172.21.15.16:0/4217576131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9de8102c40 0x7f9de806fcb0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f9dd8002a50 tx=0x7f9dd8004600 comp rx=0 tx=0).stop 2024-06-18T02:58:20.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:20.943+0000 7f9def3a8640 1 -- 172.21.15.16:0/4217576131 shutdown_connections 2024-06-18T02:58:20.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:20.943+0000 7f9def3a8640 1 --2- 172.21.15.16:0/4217576131 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f9dd4065430 0x7f9dd40678e0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:20.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:20.943+0000 7f9def3a8640 1 --2- 172.21.15.16:0/4217576131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9de8102c40 0x7f9de806fcb0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:20.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:20.943+0000 7f9def3a8640 1 --2- 172.21.15.16:0/4217576131 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9de81022a0 0x7f9de806f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:20.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:20.943+0000 7f9def3a8640 1 -- 172.21.15.16:0/4217576131 >> 172.21.15.16:0/4217576131 conn(0x7f9de80f7e50 msgr2=0x7f9de80f8db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:58:20.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:20.943+0000 7f9def3a8640 1 -- 172.21.15.16:0/4217576131 shutdown_connections 2024-06-18T02:58:20.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:20.943+0000 7f9def3a8640 1 -- 172.21.15.16:0/4217576131 wait complete. 2024-06-18T02:58:21.595 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378647 got 81604378650 for osd.2 2024-06-18T02:58:21.596 DEBUG:teuthology.parallel:result is None 2024-06-18T02:58:21.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:58:21 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/4217576131' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-06-18T02:58:21.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:58:21 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/4217576131' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-06-18T02:58:22.189 INFO:tasks.cephadm.ceph_manager.ceph:need seq 171798691849 got 171798691852 for osd.5 2024-06-18T02:58:22.189 DEBUG:teuthology.parallel:result is None 2024-06-18T02:58:22.750 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:58:22 smithi016 ceph-mon[29214]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:58:22.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:58:22 smithi123 ceph-mon[36358]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:58:23.088 INFO:tasks.cephadm.ceph_manager.ceph:need seq 115964117012 got 115964117014 for osd.3 2024-06-18T02:58:23.089 DEBUG:teuthology.parallel:result is None 2024-06-18T02:58:23.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:22.427+0000 7f3f4b7f5640 1 -- 172.21.15.16:0/2454737279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f44102c90 msgr2=0x7f3f440fa9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:23.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:22.427+0000 7f3f4b7f5640 1 --2- 172.21.15.16:0/2454737279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f44102c90 0x7f3f440fa9e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f3f340048d0 tx=0x7f3f34009da0 comp rx=0 tx=0).stop 2024-06-18T02:58:23.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:22.427+0000 7f3f4b7f5640 1 -- 172.21.15.16:0/2454737279 shutdown_connections 2024-06-18T02:58:23.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:22.427+0000 7f3f4b7f5640 1 --2- 172.21.15.16:0/2454737279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f44102c90 0x7f3f440fa9e0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:23.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:22.427+0000 7f3f4b7f5640 1 --2- 172.21.15.16:0/2454737279 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f441022f0 0x7f3f441026c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:23.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:22.427+0000 7f3f4b7f5640 1 -- 172.21.15.16:0/2454737279 >> 172.21.15.16:0/2454737279 conn(0x7f3f44070a40 msgr2=0x7f3f44070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:58:23.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:22.427+0000 7f3f4b7f5640 1 -- 172.21.15.16:0/2454737279 shutdown_connections 2024-06-18T02:58:23.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:22.427+0000 7f3f4b7f5640 1 -- 172.21.15.16:0/2454737279 wait complete. 2024-06-18T02:58:23.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:22.428+0000 7f3f4b7f5640 1 Processor -- start 2024-06-18T02:58:23.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:22.428+0000 7f3f4b7f5640 1 -- start start 2024-06-18T02:58:23.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:22.428+0000 7f3f4b7f5640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f441022f0 0x7f3f441130f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:58:23.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:22.428+0000 7f3f4b7f5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f44102c90 0x7f3f44113630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:58:23.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:22.428+0000 7f3f4b7f5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f44113c30 con 0x7f3f44102c90 2024-06-18T02:58:23.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:22.428+0000 7f3f4b7f5640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f4410d070 con 0x7f3f441022f0 2024-06-18T02:58:23.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:22.428+0000 7f3f48d90640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f44102c90 0x7f3f44113630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:58:23.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:22.428+0000 7f3f48d90640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f44102c90 0x7f3f44113630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58878/0 (socket says 172.21.15.16:58878) 2024-06-18T02:58:23.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:22.428+0000 7f3f48d90640 1 -- 172.21.15.16:0/2773036254 learned_addr learned my addr 172.21.15.16:0/2773036254 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:58:23.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:22.428+0000 7f3f49591640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f441022f0 0x7f3f441130f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:58:23.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:22.429+0000 7f3f48d90640 1 -- 172.21.15.16:0/2773036254 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f441022f0 msgr2=0x7f3f441130f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:23.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:22.429+0000 7f3f48d90640 1 --2- 172.21.15.16:0/2773036254 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f441022f0 0x7f3f441130f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:23.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:22.429+0000 7f3f48d90640 1 -- 172.21.15.16:0/2773036254 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f34005040 con 0x7f3f44102c90 2024-06-18T02:58:23.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:22.429+0000 7f3f49591640 1 --2- 172.21.15.16:0/2773036254 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f441022f0 0x7f3f441130f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-18T02:58:23.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:22.429+0000 7f3f48d90640 1 --2- 172.21.15.16:0/2773036254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f44102c90 0x7f3f44113630 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f3f34000c00 tx=0x7f3f34009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:58:23.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:22.429+0000 7f3f39ffb640 1 -- 172.21.15.16:0/2773036254 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f34038070 con 0x7f3f44102c90 2024-06-18T02:58:23.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:22.429+0000 7f3f39ffb640 1 -- 172.21.15.16:0/2773036254 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3f34032970 con 0x7f3f44102c90 2024-06-18T02:58:23.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:22.429+0000 7f3f4b7f5640 1 -- 172.21.15.16:0/2773036254 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f4410d2f0 con 0x7f3f44102c90 2024-06-18T02:58:23.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:22.429+0000 7f3f39ffb640 1 -- 172.21.15.16:0/2773036254 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f3403d980 con 0x7f3f44102c90 2024-06-18T02:58:23.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:22.429+0000 7f3f4b7f5640 1 -- 172.21.15.16:0/2773036254 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f4410d7b0 con 0x7f3f44102c90 2024-06-18T02:58:23.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:22.430+0000 7f3f4b7f5640 1 -- 172.21.15.16:0/2773036254 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f44065510 con 0x7f3f44102c90 2024-06-18T02:58:23.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:22.431+0000 7f3f39ffb640 1 -- 172.21.15.16:0/2773036254 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f3f34030020 con 0x7f3f44102c90 2024-06-18T02:58:23.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:22.431+0000 7f3f39ffb640 1 --2- 172.21.15.16:0/2773036254 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f3f30065430 0x7f3f300678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:58:23.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:22.431+0000 7f3f39ffb640 1 -- 172.21.15.16:0/2773036254 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f3f340a2520 con 0x7f3f44102c90 2024-06-18T02:58:23.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:22.431+0000 7f3f49591640 1 --2- 172.21.15.16:0/2773036254 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f3f30065430 0x7f3f300678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:58:23.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:22.432+0000 7f3f49591640 1 --2- 172.21.15.16:0/2773036254 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f3f30065430 0x7f3f300678e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3f40004fd0 tx=0x7f3f4000b040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:58:23.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:22.435+0000 7f3f39ffb640 1 -- 172.21.15.16:0/2773036254 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f3f340709d0 con 0x7f3f44102c90 2024-06-18T02:58:23.180 INFO:teuthology.orchestra.run.smithi016.stdout:38654705701 2024-06-18T02:58:23.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:22.613+0000 7f3f4b7f5640 1 -- 172.21.15.16:0/2773036254 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f3f441094a0 con 0x7f3f44102c90 2024-06-18T02:58:23.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:22.613+0000 7f3f39ffb640 1 -- 172.21.15.16:0/2773036254 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3f34073ff0 con 0x7f3f44102c90 2024-06-18T02:58:23.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:22.615+0000 7f3f4b7f5640 1 -- 172.21.15.16:0/2773036254 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f3f30065430 msgr2=0x7f3f300678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:23.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:22.615+0000 7f3f4b7f5640 1 --2- 172.21.15.16:0/2773036254 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f3f30065430 0x7f3f300678e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3f40004fd0 tx=0x7f3f4000b040 comp rx=0 tx=0).stop 2024-06-18T02:58:23.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:22.615+0000 7f3f4b7f5640 1 -- 172.21.15.16:0/2773036254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f44102c90 msgr2=0x7f3f44113630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:23.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:22.615+0000 7f3f4b7f5640 1 --2- 172.21.15.16:0/2773036254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f44102c90 0x7f3f44113630 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f3f34000c00 tx=0x7f3f34009da0 comp rx=0 tx=0).stop 2024-06-18T02:58:23.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:22.615+0000 7f3f4b7f5640 1 -- 172.21.15.16:0/2773036254 shutdown_connections 2024-06-18T02:58:23.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:22.615+0000 7f3f4b7f5640 1 --2- 172.21.15.16:0/2773036254 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f3f30065430 0x7f3f300678e0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:23.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:22.615+0000 7f3f4b7f5640 1 --2- 172.21.15.16:0/2773036254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f44102c90 0x7f3f44113630 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:23.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:22.615+0000 7f3f4b7f5640 1 --2- 172.21.15.16:0/2773036254 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3f441022f0 0x7f3f441130f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:23.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:22.615+0000 7f3f4b7f5640 1 -- 172.21.15.16:0/2773036254 >> 172.21.15.16:0/2773036254 conn(0x7f3f44070a40 msgr2=0x7f3f441050b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:58:23.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:22.616+0000 7f3f4b7f5640 1 -- 172.21.15.16:0/2773036254 shutdown_connections 2024-06-18T02:58:23.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:22.616+0000 7f3f4b7f5640 1 -- 172.21.15.16:0/2773036254 wait complete. 2024-06-18T02:58:23.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:58:23 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/2773036254' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-06-18T02:58:23.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:58:23 smithi016 ceph-mon[29214]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:58:23.881 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705699 got 38654705701 for osd.0 2024-06-18T02:58:23.881 DEBUG:teuthology.parallel:result is None 2024-06-18T02:58:23.927 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T02:58:23.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:58:23 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/2773036254' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-06-18T02:58:23.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:58:23 smithi123 ceph-mon[36358]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:58:24.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:24.595+0000 7f828adca640 1 -- 172.21.15.16:0/160241307 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82840650a0 msgr2=0x7f8284065470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:24.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:24.595+0000 7f828adca640 1 --2- 172.21.15.16:0/160241307 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82840650a0 0x7f8284065470 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f82740048d0 tx=0x7f8274009da0 comp rx=0 tx=0).stop 2024-06-18T02:58:24.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:24.595+0000 7f828adca640 1 -- 172.21.15.16:0/160241307 shutdown_connections 2024-06-18T02:58:24.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:24.595+0000 7f828adca640 1 --2- 172.21.15.16:0/160241307 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82840fdeb0 0x7f82840fe2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:24.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:24.595+0000 7f828adca640 1 --2- 172.21.15.16:0/160241307 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82840650a0 0x7f8284065470 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:24.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:24.595+0000 7f828adca640 1 -- 172.21.15.16:0/160241307 >> 172.21.15.16:0/160241307 conn(0x7f8284072110 msgr2=0x7f8284072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:58:24.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:24.595+0000 7f828adca640 1 -- 172.21.15.16:0/160241307 shutdown_connections 2024-06-18T02:58:24.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:24.595+0000 7f828adca640 1 -- 172.21.15.16:0/160241307 wait complete. 2024-06-18T02:58:24.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:24.595+0000 7f828adca640 1 Processor -- start 2024-06-18T02:58:24.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:24.595+0000 7f828adca640 1 -- start start 2024-06-18T02:58:24.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:24.596+0000 7f828adca640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82840650a0 0x7f82840fb2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:58:24.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:24.596+0000 7f828adca640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82840fdeb0 0x7f82840fb800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:58:24.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:24.596+0000 7f828adca640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82840fd070 con 0x7f82840650a0 2024-06-18T02:58:24.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:24.596+0000 7f828adca640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82840fd1e0 con 0x7f82840fdeb0 2024-06-18T02:58:24.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:24.596+0000 7f8288b66640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82840650a0 0x7f82840fb2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:58:24.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:24.596+0000 7f8288b66640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82840650a0 0x7f82840fb2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58896/0 (socket says 172.21.15.16:58896) 2024-06-18T02:58:24.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:24.596+0000 7f8288b66640 1 -- 172.21.15.16:0/3921764113 learned_addr learned my addr 172.21.15.16:0/3921764113 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:58:24.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:24.596+0000 7f8283fff640 1 --2- 172.21.15.16:0/3921764113 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82840fdeb0 0x7f82840fb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:58:24.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:24.596+0000 7f8288b66640 1 -- 172.21.15.16:0/3921764113 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82840fdeb0 msgr2=0x7f82840fb800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:24.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:24.596+0000 7f8288b66640 1 --2- 172.21.15.16:0/3921764113 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82840fdeb0 0x7f82840fb800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:24.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:24.596+0000 7f8288b66640 1 -- 172.21.15.16:0/3921764113 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8274005040 con 0x7f82840650a0 2024-06-18T02:58:24.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:24.596+0000 7f8283fff640 1 --2- 172.21.15.16:0/3921764113 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82840fdeb0 0x7f82840fb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-18T02:58:24.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:24.596+0000 7f8288b66640 1 --2- 172.21.15.16:0/3921764113 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82840650a0 0x7f82840fb2c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f8274032430 tx=0x7f8274009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:58:24.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:24.596+0000 7f82817fa640 1 -- 172.21.15.16:0/3921764113 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8274038070 con 0x7f82840650a0 2024-06-18T02:58:24.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:24.597+0000 7f82817fa640 1 -- 172.21.15.16:0/3921764113 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8274008ee0 con 0x7f82840650a0 2024-06-18T02:58:24.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:24.597+0000 7f82817fa640 1 -- 172.21.15.16:0/3921764113 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f827403d7a0 con 0x7f82840650a0 2024-06-18T02:58:24.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:24.597+0000 7f828adca640 1 -- 172.21.15.16:0/3921764113 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82840fbda0 con 0x7f82840650a0 2024-06-18T02:58:24.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:24.597+0000 7f828adca640 1 -- 172.21.15.16:0/3921764113 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f828411db10 con 0x7f82840650a0 2024-06-18T02:58:24.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:24.598+0000 7f828adca640 1 -- 172.21.15.16:0/3921764113 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82840fda50 con 0x7f82840650a0 2024-06-18T02:58:24.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:24.598+0000 7f82817fa640 1 -- 172.21.15.16:0/3921764113 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f8274030020 con 0x7f82840650a0 2024-06-18T02:58:24.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:24.598+0000 7f82817fa640 1 --2- 172.21.15.16:0/3921764113 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f826c065430 0x7f826c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:58:24.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:24.598+0000 7f82817fa640 1 -- 172.21.15.16:0/3921764113 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f82740a2b20 con 0x7f82840650a0 2024-06-18T02:58:24.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:24.599+0000 7f8283fff640 1 --2- 172.21.15.16:0/3921764113 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f826c065430 0x7f826c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:58:24.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:24.599+0000 7f8283fff640 1 --2- 172.21.15.16:0/3921764113 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f826c065430 0x7f826c0678e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f82840fcc00 tx=0x7f8278009040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:58:24.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:24.603+0000 7f82817fa640 1 -- 172.21.15.16:0/3921764113 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f8274071090 con 0x7f82840650a0 2024-06-18T02:58:24.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:24.781+0000 7f828adca640 1 -- 172.21.15.16:0/3921764113 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f82840fc9b0 con 0x7f82840650a0 2024-06-18T02:58:24.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:24.781+0000 7f82817fa640 1 -- 172.21.15.16:0/3921764113 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f82740746b0 con 0x7f82840650a0 2024-06-18T02:58:24.782 INFO:teuthology.orchestra.run.smithi016.stdout:150323855378 2024-06-18T02:58:24.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:24.783+0000 7f828adca640 1 -- 172.21.15.16:0/3921764113 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f826c065430 msgr2=0x7f826c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:24.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:24.783+0000 7f828adca640 1 --2- 172.21.15.16:0/3921764113 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f826c065430 0x7f826c0678e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f82840fcc00 tx=0x7f8278009040 comp rx=0 tx=0).stop 2024-06-18T02:58:24.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:24.783+0000 7f828adca640 1 -- 172.21.15.16:0/3921764113 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82840650a0 msgr2=0x7f82840fb2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:24.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:24.783+0000 7f828adca640 1 --2- 172.21.15.16:0/3921764113 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82840650a0 0x7f82840fb2c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f8274032430 tx=0x7f8274009da0 comp rx=0 tx=0).stop 2024-06-18T02:58:24.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:24.783+0000 7f828adca640 1 -- 172.21.15.16:0/3921764113 shutdown_connections 2024-06-18T02:58:24.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:24.784+0000 7f828adca640 1 --2- 172.21.15.16:0/3921764113 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f826c065430 0x7f826c0678e0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:24.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:24.784+0000 7f828adca640 1 --2- 172.21.15.16:0/3921764113 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82840fdeb0 0x7f82840fb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:24.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:24.784+0000 7f828adca640 1 --2- 172.21.15.16:0/3921764113 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82840650a0 0x7f82840fb2c0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:24.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:24.784+0000 7f828adca640 1 -- 172.21.15.16:0/3921764113 >> 172.21.15.16:0/3921764113 conn(0x7f8284072110 msgr2=0x7f82840f8610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:58:24.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:24.784+0000 7f828adca640 1 -- 172.21.15.16:0/3921764113 shutdown_connections 2024-06-18T02:58:24.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:24.784+0000 7f828adca640 1 -- 172.21.15.16:0/3921764113 wait complete. 2024-06-18T02:58:25.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:58:24 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/3921764113' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-06-18T02:58:25.234 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:58:24 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/3921764113' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-06-18T02:58:26.139 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:58:25 smithi016 ceph-mon[29214]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:58:26.170 INFO:tasks.cephadm.ceph_manager.ceph:need seq 150323855375 got 150323855378 for osd.4 2024-06-18T02:58:26.170 DEBUG:teuthology.parallel:result is None 2024-06-18T02:58:26.234 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:58:25 smithi123 ceph-mon[36358]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:58:26.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:26.634+0000 7fec1f279640 1 -- 172.21.15.16:0/2591536641 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec181022f0 msgr2=0x7fec181026c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:26.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:26.634+0000 7fec1f279640 1 --2- 172.21.15.16:0/2591536641 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec181022f0 0x7fec181026c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fec080048d0 tx=0x7fec08009da0 comp rx=0 tx=0).stop 2024-06-18T02:58:26.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:26.634+0000 7fec1f279640 1 -- 172.21.15.16:0/2591536641 shutdown_connections 2024-06-18T02:58:26.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:26.634+0000 7fec1f279640 1 --2- 172.21.15.16:0/2591536641 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fec18102c90 0x7fec180fc320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:26.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:26.634+0000 7fec1f279640 1 --2- 172.21.15.16:0/2591536641 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec181022f0 0x7fec181026c0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:26.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:26.634+0000 7fec1f279640 1 -- 172.21.15.16:0/2591536641 >> 172.21.15.16:0/2591536641 conn(0x7fec180f7f00 msgr2=0x7fec180fa310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:58:26.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:26.635+0000 7fec1f279640 1 -- 172.21.15.16:0/2591536641 shutdown_connections 2024-06-18T02:58:26.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:26.635+0000 7fec1f279640 1 -- 172.21.15.16:0/2591536641 wait complete. 2024-06-18T02:58:26.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:26.635+0000 7fec1f279640 1 Processor -- start 2024-06-18T02:58:26.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:26.635+0000 7fec1f279640 1 -- start start 2024-06-18T02:58:26.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:26.635+0000 7fec1f279640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fec181022f0 0x7fec1806f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:58:26.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:26.635+0000 7fec1f279640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec18102c90 0x7fec1806fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:58:26.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:26.635+0000 7fec1f279640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec180701f0 con 0x7fec18102c90 2024-06-18T02:58:26.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:26.635+0000 7fec1f279640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec18070360 con 0x7fec181022f0 2024-06-18T02:58:26.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:26.636+0000 7fec1c814640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec18102c90 0x7fec1806fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:58:26.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:26.636+0000 7fec1c814640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec18102c90 0x7fec1806fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58914/0 (socket says 172.21.15.16:58914) 2024-06-18T02:58:26.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:26.636+0000 7fec1c814640 1 -- 172.21.15.16:0/2869517465 learned_addr learned my addr 172.21.15.16:0/2869517465 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:58:26.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:26.636+0000 7fec1d015640 1 --2- 172.21.15.16:0/2869517465 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fec181022f0 0x7fec1806f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:58:26.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:26.636+0000 7fec1c814640 1 -- 172.21.15.16:0/2869517465 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fec181022f0 msgr2=0x7fec1806f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:26.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:26.636+0000 7fec1c814640 1 --2- 172.21.15.16:0/2869517465 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fec181022f0 0x7fec1806f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:26.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:26.636+0000 7fec1c814640 1 -- 172.21.15.16:0/2869517465 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec08005040 con 0x7fec18102c90 2024-06-18T02:58:26.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:26.636+0000 7fec1c814640 1 --2- 172.21.15.16:0/2869517465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec18102c90 0x7fec1806fcb0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fec14002a50 tx=0x7fec14004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:58:26.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:26.636+0000 7fec0dffb640 1 -- 172.21.15.16:0/2869517465 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fec14007c30 con 0x7fec18102c90 2024-06-18T02:58:26.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:26.636+0000 7fec0dffb640 1 -- 172.21.15.16:0/2869517465 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fec1400f040 con 0x7fec18102c90 2024-06-18T02:58:26.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:26.636+0000 7fec1f279640 1 -- 172.21.15.16:0/2869517465 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec180705f0 con 0x7fec18102c90 2024-06-18T02:58:26.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:26.636+0000 7fec0dffb640 1 -- 172.21.15.16:0/2869517465 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fec1400dad0 con 0x7fec18102c90 2024-06-18T02:58:26.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:26.636+0000 7fec1f279640 1 -- 172.21.15.16:0/2869517465 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec18119720 con 0x7fec18102c90 2024-06-18T02:58:26.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:26.637+0000 7fec1f279640 1 -- 172.21.15.16:0/2869517465 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec180fda60 con 0x7fec18102c90 2024-06-18T02:58:26.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:26.638+0000 7fec0dffb640 1 -- 172.21.15.16:0/2869517465 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fec1400dd40 con 0x7fec18102c90 2024-06-18T02:58:26.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:26.638+0000 7fec0dffb640 1 --2- 172.21.15.16:0/2869517465 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fec04065430 0x7fec040678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:58:26.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:26.638+0000 7fec0dffb640 1 -- 172.21.15.16:0/2869517465 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fec14081c90 con 0x7fec18102c90 2024-06-18T02:58:26.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:26.638+0000 7fec1d015640 1 --2- 172.21.15.16:0/2869517465 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fec04065430 0x7fec040678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:58:26.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:26.639+0000 7fec1d015640 1 --2- 172.21.15.16:0/2869517465 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fec04065430 0x7fec040678e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fec08009f80 tx=0x7fec08035040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:58:26.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:26.642+0000 7fec0dffb640 1 -- 172.21.15.16:0/2869517465 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fec1404cdf0 con 0x7fec18102c90 2024-06-18T02:58:26.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:26.818+0000 7fec1f279640 1 -- 172.21.15.16:0/2869517465 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fec180fbb00 con 0x7fec18102c90 2024-06-18T02:58:26.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:26.818+0000 7fec0dffb640 1 -- 172.21.15.16:0/2869517465 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fec140501f0 con 0x7fec18102c90 2024-06-18T02:58:26.819 INFO:teuthology.orchestra.run.smithi016.stdout:60129542178 2024-06-18T02:58:26.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:26.820+0000 7fec1f279640 1 -- 172.21.15.16:0/2869517465 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fec04065430 msgr2=0x7fec040678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:26.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:26.820+0000 7fec1f279640 1 --2- 172.21.15.16:0/2869517465 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fec04065430 0x7fec040678e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fec08009f80 tx=0x7fec08035040 comp rx=0 tx=0).stop 2024-06-18T02:58:26.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:26.820+0000 7fec1f279640 1 -- 172.21.15.16:0/2869517465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec18102c90 msgr2=0x7fec1806fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:26.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:26.820+0000 7fec1f279640 1 --2- 172.21.15.16:0/2869517465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec18102c90 0x7fec1806fcb0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fec14002a50 tx=0x7fec14004600 comp rx=0 tx=0).stop 2024-06-18T02:58:26.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:26.820+0000 7fec1f279640 1 -- 172.21.15.16:0/2869517465 shutdown_connections 2024-06-18T02:58:26.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:26.820+0000 7fec1f279640 1 --2- 172.21.15.16:0/2869517465 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fec04065430 0x7fec040678e0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:26.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:26.820+0000 7fec1f279640 1 --2- 172.21.15.16:0/2869517465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec18102c90 0x7fec1806fcb0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:26.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:26.820+0000 7fec1f279640 1 --2- 172.21.15.16:0/2869517465 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fec181022f0 0x7fec1806f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:26.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:26.820+0000 7fec1f279640 1 -- 172.21.15.16:0/2869517465 >> 172.21.15.16:0/2869517465 conn(0x7fec180f7f00 msgr2=0x7fec180f8e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:58:26.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:26.820+0000 7fec1f279640 1 -- 172.21.15.16:0/2869517465 shutdown_connections 2024-06-18T02:58:26.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:26.820+0000 7fec1f279640 1 -- 172.21.15.16:0/2869517465 wait complete. 2024-06-18T02:58:26.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:58:26 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:58:26.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:58:26 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/2869517465' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-06-18T02:58:27.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:58:26 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:58:27.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:58:26 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/2869517465' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-06-18T02:58:27.590 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542175 got 60129542178 for osd.1 2024-06-18T02:58:27.591 DEBUG:teuthology.parallel:result is None 2024-06-18T02:58:27.591 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-06-18T02:58:27.591 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph pg dump --format=json 2024-06-18T02:58:27.860 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T02:58:28.114 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:58:27 smithi016 ceph-mon[29214]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:58:28.234 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:58:27 smithi123 ceph-mon[36358]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:58:29.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:29.466+0000 7f135254b640 1 -- 172.21.15.16:0/3983446154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f134c1022f0 msgr2=0x7f134c1026c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:29.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:29.466+0000 7f135254b640 1 --2- 172.21.15.16:0/3983446154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f134c1022f0 0x7f134c1026c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f133c0048d0 tx=0x7f133c009da0 comp rx=0 tx=0).stop 2024-06-18T02:58:29.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:29.466+0000 7f135254b640 1 -- 172.21.15.16:0/3983446154 shutdown_connections 2024-06-18T02:58:29.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:29.466+0000 7f135254b640 1 --2- 172.21.15.16:0/3983446154 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f134c102c90 0x7f134c0fa9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:29.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:29.466+0000 7f135254b640 1 --2- 172.21.15.16:0/3983446154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f134c1022f0 0x7f134c1026c0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:29.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:29.466+0000 7f135254b640 1 -- 172.21.15.16:0/3983446154 >> 172.21.15.16:0/3983446154 conn(0x7f134c070a40 msgr2=0x7f134c070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:58:29.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:29.466+0000 7f135254b640 1 -- 172.21.15.16:0/3983446154 shutdown_connections 2024-06-18T02:58:29.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:29.466+0000 7f135254b640 1 -- 172.21.15.16:0/3983446154 wait complete. 2024-06-18T02:58:29.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:29.467+0000 7f135254b640 1 Processor -- start 2024-06-18T02:58:29.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:29.467+0000 7f135254b640 1 -- start start 2024-06-18T02:58:29.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:29.467+0000 7f135254b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f134c1022f0 0x7f134c113130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:58:29.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:29.467+0000 7f135254b640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f134c102c90 0x7f134c113670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:58:29.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:29.467+0000 7f135254b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f134c113c70 con 0x7f134c1022f0 2024-06-18T02:58:29.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:29.467+0000 7f135254b640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f134c10d0b0 con 0x7f134c102c90 2024-06-18T02:58:29.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:29.467+0000 7f134bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f134c1022f0 0x7f134c113130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:58:29.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:29.467+0000 7f134bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f134c1022f0 0x7f134c113130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58944/0 (socket says 172.21.15.16:58944) 2024-06-18T02:58:29.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:29.467+0000 7f134bfff640 1 -- 172.21.15.16:0/1181225082 learned_addr learned my addr 172.21.15.16:0/1181225082 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:58:29.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:29.467+0000 7f134b7fe640 1 --2- 172.21.15.16:0/1181225082 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f134c102c90 0x7f134c113670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:58:29.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:29.467+0000 7f134bfff640 1 -- 172.21.15.16:0/1181225082 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f134c102c90 msgr2=0x7f134c113670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:29.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:29.467+0000 7f134bfff640 1 --2- 172.21.15.16:0/1181225082 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f134c102c90 0x7f134c113670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:29.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:29.467+0000 7f134bfff640 1 -- 172.21.15.16:0/1181225082 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f133c005040 con 0x7f134c1022f0 2024-06-18T02:58:29.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:29.467+0000 7f134b7fe640 1 --2- 172.21.15.16:0/1181225082 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f134c102c90 0x7f134c113670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-18T02:58:29.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:29.468+0000 7f134bfff640 1 --2- 172.21.15.16:0/1181225082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f134c1022f0 0x7f134c113130 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f133c0043d0 tx=0x7f133c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:58:29.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:29.468+0000 7f1348ff9640 1 -- 172.21.15.16:0/1181225082 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f133c038070 con 0x7f134c1022f0 2024-06-18T02:58:29.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:29.468+0000 7f135254b640 1 -- 172.21.15.16:0/1181225082 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f134c10d330 con 0x7f134c1022f0 2024-06-18T02:58:29.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:29.468+0000 7f1348ff9640 1 -- 172.21.15.16:0/1181225082 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f133c02abd0 con 0x7f134c1022f0 2024-06-18T02:58:29.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:29.468+0000 7f1348ff9640 1 -- 172.21.15.16:0/1181225082 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f133c033640 con 0x7f134c1022f0 2024-06-18T02:58:29.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:29.468+0000 7f135254b640 1 -- 172.21.15.16:0/1181225082 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f134c10d790 con 0x7f134c1022f0 2024-06-18T02:58:29.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:29.469+0000 7f135254b640 1 -- 172.21.15.16:0/1181225082 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f134c065510 con 0x7f134c1022f0 2024-06-18T02:58:29.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:29.469+0000 7f1348ff9640 1 -- 172.21.15.16:0/1181225082 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f133c030020 con 0x7f134c1022f0 2024-06-18T02:58:29.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:29.470+0000 7f1348ff9640 1 --2- 172.21.15.16:0/1181225082 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f1334065430 0x7f13340678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:58:29.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:29.470+0000 7f1348ff9640 1 -- 172.21.15.16:0/1181225082 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f133c0a22d0 con 0x7f134c1022f0 2024-06-18T02:58:29.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:29.470+0000 7f134b7fe640 1 --2- 172.21.15.16:0/1181225082 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f1334065430 0x7f13340678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:58:29.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:29.470+0000 7f134b7fe640 1 --2- 172.21.15.16:0/1181225082 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f1334065430 0x7f13340678e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f134c10e9f0 tx=0x7f134000b040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:58:29.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:29.474+0000 7f1348ff9640 1 -- 172.21.15.16:0/1181225082 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f133c070780 con 0x7f134c1022f0 2024-06-18T02:58:29.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:29.644+0000 7f135254b640 1 -- 172.21.15.16:0/1181225082 --> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f134c04b480 con 0x7f1334065430 2024-06-18T02:58:29.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:29.645+0000 7f1348ff9640 1 -- 172.21.15.16:0/1181225082 <== mgr.14216 v2:172.21.15.16:6800/3814182425 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+28096 (secure 0 0 0) 0x7f134c04b480 con 0x7f1334065430 2024-06-18T02:58:29.646 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:58:29.646 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-06-18T02:58:29.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:29.647+0000 7f135254b640 1 -- 172.21.15.16:0/1181225082 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f1334065430 msgr2=0x7f13340678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:29.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:29.647+0000 7f135254b640 1 --2- 172.21.15.16:0/1181225082 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f1334065430 0x7f13340678e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f134c10e9f0 tx=0x7f134000b040 comp rx=0 tx=0).stop 2024-06-18T02:58:29.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:29.647+0000 7f135254b640 1 -- 172.21.15.16:0/1181225082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f134c1022f0 msgr2=0x7f134c113130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:29.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:29.647+0000 7f135254b640 1 --2- 172.21.15.16:0/1181225082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f134c1022f0 0x7f134c113130 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f133c0043d0 tx=0x7f133c009da0 comp rx=0 tx=0).stop 2024-06-18T02:58:29.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:29.647+0000 7f135254b640 1 -- 172.21.15.16:0/1181225082 shutdown_connections 2024-06-18T02:58:29.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:29.647+0000 7f135254b640 1 --2- 172.21.15.16:0/1181225082 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f1334065430 0x7f13340678e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:29.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:29.647+0000 7f135254b640 1 --2- 172.21.15.16:0/1181225082 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f134c102c90 0x7f134c113670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:29.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:29.647+0000 7f135254b640 1 --2- 172.21.15.16:0/1181225082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f134c1022f0 0x7f134c113130 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:29.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:29.647+0000 7f135254b640 1 -- 172.21.15.16:0/1181225082 >> 172.21.15.16:0/1181225082 conn(0x7f134c070a40 msgr2=0x7f134c1050b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:58:29.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:29.647+0000 7f135254b640 1 -- 172.21.15.16:0/1181225082 shutdown_connections 2024-06-18T02:58:29.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:29.647+0000 7f135254b640 1 -- 172.21.15.16:0/1181225082 wait complete. 2024-06-18T02:58:30.363 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":158,"stamp":"2024-06-18T02:58:29.265389+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1788068,"kb_used_data":3924,"kb_used_omap":9,"kb_used_meta":1784054,"kb_avail":560682844,"statfs":{"total":575970213888,"available":574139232256,"internally_reserved":0,"allocated":4018176,"data_stored":2624466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1826871312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002298"},"pg_stats":[{"pgid":"1.0","version":"24'117","reported_seq":187,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-18T02:57:35.451474+0000","last_change":"2024-06-18T02:57:10.349620+0000","last_active":"2024-06-18T02:57:35.451474+0000","last_peered":"2024-06-18T02:57:35.451474+0000","last_clean":"2024-06-18T02:57:35.451474+0000","last_became_active":"2024-06-18T02:57:10.349190+0000","last_became_peered":"2024-06-18T02:57:10.349190+0000","last_unstale":"2024-06-18T02:57:35.451474+0000","last_undegraded":"2024-06-18T02:57:35.451474+0000","last_fullsized":"2024-06-18T02:57:35.451474+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-18T02:56:23.754385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-18T02:56:23.754385+0000","last_clean_scrub_stamp":"2024-06-18T02:56:23.754385+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-19T05:25:07.885787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":40,"seq":171798691854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297720,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447432,"statfs":{"total":95995035648,"available":95690170368,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41599999999999998}]}]},{"osd":4,"up_from":35,"seq":150323855379,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297724,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447428,"statfs":{"total":95995035648,"available":95690166272,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jun 18 02:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.441},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.42899999999999999}]},{"osd":1,"last update":"Tue Jun 18 02:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.40799999999999997}]},{"osd":2,"last update":"Tue Jun 18 02:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.45600000000000002}]},{"osd":3,"last update":"Tue Jun 18 02:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.622,"5min":0.622,"15min":0.622},"last":0.34300000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.379}]}]},{"osd":3,"up_from":27,"seq":115964117016,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298300,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446852,"statfs":{"total":95995035648,"available":95689576448,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jun 18 02:57:46 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.40100000000000002}]},{"osd":1,"last update":"Tue Jun 18 02:57:46 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.50800000000000001}]},{"osd":2,"last update":"Tue Jun 18 02:57:46 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.44700000000000001}]},{"osd":4,"last update":"Tue Jun 18 02:58:10 2024","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"last":0.58399999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]}]},{"osd":2,"up_from":19,"seq":81604378652,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297720,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447432,"statfs":{"total":95995035648,"available":95690170368,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jun 18 02:58:26 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.36799999999999999,"15min":0.37},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.51100000000000001,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.377,"15min":0.38100000000000001},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.52700000000000002,"5min":0.55800000000000005,"15min":0.55800000000000005},"last":0.33400000000000002}]},{"osd":1,"last update":"Tue Jun 18 02:58:26 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.38900000000000001,"15min":0.39400000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"last":0.317},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.39000000000000001,"15min":0.39500000000000002},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.54700000000000004,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.38100000000000001}]},{"osd":3,"last update":"Tue Jun 18 02:57:44 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.49099999999999999}]},{"osd":4,"last update":"Tue Jun 18 02:58:13 2024","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.3029999999999999,"15min":1.3029999999999999},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":19.146000000000001,"5min":19.146000000000001,"15min":19.146000000000001},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.45400000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48299999999999998}]}]},{"osd":1,"up_from":14,"seq":60129542179,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298300,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446852,"statfs":{"total":95995035648,"available":95689576448,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jun 18 02:57:54 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.33300000000000002,"15min":0.32500000000000001},"min":{"1min":0.22800000000000001,"5min":0.214,"15min":0.214},"max":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.377,"5min":0.32800000000000001,"15min":0.32000000000000001},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"last":0.372}]},{"osd":2,"last update":"Tue Jun 18 02:57:26 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.377,"15min":0.377},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.627,"5min":0.627,"15min":0.627},"last":0.30099999999999999}]},{"osd":3,"last update":"Tue Jun 18 02:57:48 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.57399999999999995}]},{"osd":4,"last update":"Tue Jun 18 02:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.67900000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]}]},{"osd":0,"up_from":9,"seq":38654705703,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298304,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446848,"statfs":{"total":95995035648,"available":95689572352,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jun 18 02:57:57 2024","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.318,"15min":0.312},"min":{"1min":0.22,"5min":0.19,"15min":0.19},"max":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.33500000000000002,"15min":0.33000000000000002},"min":{"1min":0.254,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"last":0.34000000000000002}]},{"osd":2,"last update":"Tue Jun 18 02:58:24 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.375,"15min":0.379},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.39900000000000002,"15min":0.40799999999999997},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.53200000000000003,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":0.42999999999999999}]},{"osd":3,"last update":"Tue Jun 18 02:57:47 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.66100000000000003}]},{"osd":4,"last update":"Tue Jun 18 02:58:11 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.72399999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-18T02:58:30.364 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph pg dump --format=json 2024-06-18T02:58:30.630 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T02:58:30.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:58:30 smithi123 ceph-mon[36358]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:58:30.753 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:58:30 smithi016 ceph-mon[29214]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:58:31.442 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:58:31 smithi016 ceph-mon[29214]: from='client.14426 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-18T02:58:31.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:58:31 smithi123 ceph-mon[36358]: from='client.14426 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-18T02:58:32.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:32.194+0000 7fab268bf640 1 -- 172.21.15.16:0/1313253891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab20102c90 msgr2=0x7fab200fa9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:32.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:32.194+0000 7fab268bf640 1 --2- 172.21.15.16:0/1313253891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab20102c90 0x7fab200fa9e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fab140048d0 tx=0x7fab14009da0 comp rx=0 tx=0).stop 2024-06-18T02:58:32.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:32.194+0000 7fab268bf640 1 -- 172.21.15.16:0/1313253891 shutdown_connections 2024-06-18T02:58:32.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:32.194+0000 7fab268bf640 1 --2- 172.21.15.16:0/1313253891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab20102c90 0x7fab200fa9e0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:32.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:32.194+0000 7fab268bf640 1 --2- 172.21.15.16:0/1313253891 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fab201022f0 0x7fab201026c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:32.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:32.194+0000 7fab268bf640 1 -- 172.21.15.16:0/1313253891 >> 172.21.15.16:0/1313253891 conn(0x7fab20070a40 msgr2=0x7fab20070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:58:32.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:32.194+0000 7fab268bf640 1 -- 172.21.15.16:0/1313253891 shutdown_connections 2024-06-18T02:58:32.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:32.194+0000 7fab268bf640 1 -- 172.21.15.16:0/1313253891 wait complete. 2024-06-18T02:58:32.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:32.194+0000 7fab268bf640 1 Processor -- start 2024-06-18T02:58:32.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:32.195+0000 7fab268bf640 1 -- start start 2024-06-18T02:58:32.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:32.195+0000 7fab268bf640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab201022f0 0x7fab200fbbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:58:32.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:32.195+0000 7fab268bf640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fab20102c90 0x7fab200fc100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:58:32.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:32.195+0000 7fab268bf640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab200fc640 con 0x7fab201022f0 2024-06-18T02:58:32.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:32.195+0000 7fab268bf640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab200fc7b0 con 0x7fab20102c90 2024-06-18T02:58:32.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:32.195+0000 7fab1ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab201022f0 0x7fab200fbbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:58:32.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:32.195+0000 7fab1ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab201022f0 0x7fab200fbbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48396/0 (socket says 172.21.15.16:48396) 2024-06-18T02:58:32.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:32.195+0000 7fab1ffff640 1 -- 172.21.15.16:0/260441455 learned_addr learned my addr 172.21.15.16:0/260441455 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:58:32.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:32.195+0000 7fab1f7fe640 1 --2- 172.21.15.16:0/260441455 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fab20102c90 0x7fab200fc100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:58:32.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:32.195+0000 7fab1ffff640 1 -- 172.21.15.16:0/260441455 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fab20102c90 msgr2=0x7fab200fc100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:32.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:32.195+0000 7fab1ffff640 1 --2- 172.21.15.16:0/260441455 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fab20102c90 0x7fab200fc100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:32.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:32.195+0000 7fab1ffff640 1 -- 172.21.15.16:0/260441455 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab14005040 con 0x7fab201022f0 2024-06-18T02:58:32.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:32.196+0000 7fab1ffff640 1 --2- 172.21.15.16:0/260441455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab201022f0 0x7fab200fbbc0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fab1000c7e0 tx=0x7fab100043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:58:32.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:32.196+0000 7fab1cff9640 1 -- 172.21.15.16:0/260441455 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab1000a7a0 con 0x7fab201022f0 2024-06-18T02:58:32.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:32.196+0000 7fab1cff9640 1 -- 172.21.15.16:0/260441455 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fab1000add0 con 0x7fab201022f0 2024-06-18T02:58:32.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:32.196+0000 7fab268bf640 1 -- 172.21.15.16:0/260441455 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab200fca40 con 0x7fab201022f0 2024-06-18T02:58:32.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:32.196+0000 7fab1cff9640 1 -- 172.21.15.16:0/260441455 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab100113a0 con 0x7fab201022f0 2024-06-18T02:58:32.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:32.196+0000 7fab268bf640 1 -- 172.21.15.16:0/260441455 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab20119770 con 0x7fab201022f0 2024-06-18T02:58:32.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:32.197+0000 7fab268bf640 1 -- 172.21.15.16:0/260441455 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab20065510 con 0x7fab201022f0 2024-06-18T02:58:32.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:32.197+0000 7fab1cff9640 1 -- 172.21.15.16:0/260441455 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fab1000a900 con 0x7fab201022f0 2024-06-18T02:58:32.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:32.198+0000 7fab1cff9640 1 --2- 172.21.15.16:0/260441455 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fab08065430 0x7fab080678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:58:32.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:32.198+0000 7fab1cff9640 1 -- 172.21.15.16:0/260441455 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fab10082420 con 0x7fab201022f0 2024-06-18T02:58:32.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:32.198+0000 7fab1f7fe640 1 --2- 172.21.15.16:0/260441455 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fab08065430 0x7fab080678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:58:32.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:32.198+0000 7fab1f7fe640 1 --2- 172.21.15.16:0/260441455 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fab08065430 0x7fab080678e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fab14000c00 tx=0x7fab14035040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:58:32.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:32.202+0000 7fab1cff9640 1 -- 172.21.15.16:0/260441455 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fab100509f0 con 0x7fab201022f0 2024-06-18T02:58:32.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:32.378+0000 7fab268bf640 1 -- 172.21.15.16:0/260441455 --> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fab200fd2b0 con 0x7fab08065430 2024-06-18T02:58:32.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:32.379+0000 7fab1cff9640 1 -- 172.21.15.16:0/260441455 <== mgr.14216 v2:172.21.15.16:6800/3814182425 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+28096 (secure 0 0 0) 0x7fab200fd2b0 con 0x7fab08065430 2024-06-18T02:58:32.380 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:58:32.380 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-06-18T02:58:32.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:32.381+0000 7fab268bf640 1 -- 172.21.15.16:0/260441455 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fab08065430 msgr2=0x7fab080678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:32.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:32.381+0000 7fab268bf640 1 --2- 172.21.15.16:0/260441455 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fab08065430 0x7fab080678e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fab14000c00 tx=0x7fab14035040 comp rx=0 tx=0).stop 2024-06-18T02:58:32.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:32.381+0000 7fab268bf640 1 -- 172.21.15.16:0/260441455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab201022f0 msgr2=0x7fab200fbbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:32.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:32.381+0000 7fab268bf640 1 --2- 172.21.15.16:0/260441455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab201022f0 0x7fab200fbbc0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fab1000c7e0 tx=0x7fab100043c0 comp rx=0 tx=0).stop 2024-06-18T02:58:32.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:32.381+0000 7fab268bf640 1 -- 172.21.15.16:0/260441455 shutdown_connections 2024-06-18T02:58:32.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:32.381+0000 7fab268bf640 1 --2- 172.21.15.16:0/260441455 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fab08065430 0x7fab080678e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:32.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:32.381+0000 7fab268bf640 1 --2- 172.21.15.16:0/260441455 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fab20102c90 0x7fab200fc100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:32.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:32.381+0000 7fab268bf640 1 --2- 172.21.15.16:0/260441455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab201022f0 0x7fab200fbbc0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:32.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:32.381+0000 7fab268bf640 1 -- 172.21.15.16:0/260441455 >> 172.21.15.16:0/260441455 conn(0x7fab20070a40 msgr2=0x7fab201050b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:58:32.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:32.382+0000 7fab268bf640 1 -- 172.21.15.16:0/260441455 shutdown_connections 2024-06-18T02:58:32.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:32.382+0000 7fab268bf640 1 -- 172.21.15.16:0/260441455 wait complete. 2024-06-18T02:58:32.682 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:58:32 smithi016 ceph-mon[29214]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:58:32.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:58:32 smithi123 ceph-mon[36358]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:58:33.035 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":159,"stamp":"2024-06-18T02:58:31.265695+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1788068,"kb_used_data":3924,"kb_used_omap":9,"kb_used_meta":1784054,"kb_avail":560682844,"statfs":{"total":575970213888,"available":574139232256,"internally_reserved":0,"allocated":4018176,"data_stored":2624466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1826871312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002307"},"pg_stats":[{"pgid":"1.0","version":"24'117","reported_seq":187,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-18T02:57:35.451474+0000","last_change":"2024-06-18T02:57:10.349620+0000","last_active":"2024-06-18T02:57:35.451474+0000","last_peered":"2024-06-18T02:57:35.451474+0000","last_clean":"2024-06-18T02:57:35.451474+0000","last_became_active":"2024-06-18T02:57:10.349190+0000","last_became_peered":"2024-06-18T02:57:10.349190+0000","last_unstale":"2024-06-18T02:57:35.451474+0000","last_undegraded":"2024-06-18T02:57:35.451474+0000","last_fullsized":"2024-06-18T02:57:35.451474+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-18T02:56:23.754385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-18T02:56:23.754385+0000","last_clean_scrub_stamp":"2024-06-18T02:56:23.754385+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-19T05:25:07.885787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":40,"seq":171798691854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297720,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447432,"statfs":{"total":95995035648,"available":95690170368,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41599999999999998}]}]},{"osd":4,"up_from":35,"seq":150323855379,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297724,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447428,"statfs":{"total":95995035648,"available":95690166272,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jun 18 02:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.441},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.42899999999999999}]},{"osd":1,"last update":"Tue Jun 18 02:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.40799999999999997}]},{"osd":2,"last update":"Tue Jun 18 02:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.45600000000000002}]},{"osd":3,"last update":"Tue Jun 18 02:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.622,"5min":0.622,"15min":0.622},"last":0.34300000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.379}]}]},{"osd":3,"up_from":27,"seq":115964117017,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298300,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446852,"statfs":{"total":95995035648,"available":95689576448,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jun 18 02:57:46 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.40100000000000002}]},{"osd":1,"last update":"Tue Jun 18 02:57:46 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.50800000000000001}]},{"osd":2,"last update":"Tue Jun 18 02:57:46 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.44700000000000001}]},{"osd":4,"last update":"Tue Jun 18 02:58:10 2024","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"last":0.58399999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]}]},{"osd":2,"up_from":19,"seq":81604378652,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297720,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447432,"statfs":{"total":95995035648,"available":95690170368,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jun 18 02:58:26 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.36799999999999999,"15min":0.37},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.51100000000000001,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.377,"15min":0.38100000000000001},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.52700000000000002,"5min":0.55800000000000005,"15min":0.55800000000000005},"last":0.33400000000000002}]},{"osd":1,"last update":"Tue Jun 18 02:58:26 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.38900000000000001,"15min":0.39400000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"last":0.317},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.39000000000000001,"15min":0.39500000000000002},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.54700000000000004,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.38100000000000001}]},{"osd":3,"last update":"Tue Jun 18 02:57:44 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.49099999999999999}]},{"osd":4,"last update":"Tue Jun 18 02:58:13 2024","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.3029999999999999,"15min":1.3029999999999999},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":19.146000000000001,"5min":19.146000000000001,"15min":19.146000000000001},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.45400000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48299999999999998}]}]},{"osd":1,"up_from":14,"seq":60129542179,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298300,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446852,"statfs":{"total":95995035648,"available":95689576448,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jun 18 02:57:54 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.33300000000000002,"15min":0.32500000000000001},"min":{"1min":0.22800000000000001,"5min":0.214,"15min":0.214},"max":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.377,"5min":0.32800000000000001,"15min":0.32000000000000001},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"last":0.372}]},{"osd":2,"last update":"Tue Jun 18 02:57:26 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.377,"15min":0.377},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.627,"5min":0.627,"15min":0.627},"last":0.30099999999999999}]},{"osd":3,"last update":"Tue Jun 18 02:57:48 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.57399999999999995}]},{"osd":4,"last update":"Tue Jun 18 02:58:12 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.67900000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]}]},{"osd":0,"up_from":9,"seq":38654705703,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298304,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446848,"statfs":{"total":95995035648,"available":95689572352,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jun 18 02:57:57 2024","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.318,"15min":0.312},"min":{"1min":0.22,"5min":0.19,"15min":0.19},"max":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.33500000000000002,"15min":0.33000000000000002},"min":{"1min":0.254,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"last":0.34000000000000002}]},{"osd":2,"last update":"Tue Jun 18 02:58:24 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.375,"15min":0.379},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.39900000000000002,"15min":0.40799999999999997},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.53200000000000003,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":0.42999999999999999}]},{"osd":3,"last update":"Tue Jun 18 02:57:47 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.66100000000000003}]},{"osd":4,"last update":"Tue Jun 18 02:58:11 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.72399999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-18T02:58:33.036 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-06-18T02:58:33.036 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-06-18T02:58:33.036 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-06-18T02:58:33.036 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T02:58:33.304 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T02:58:33.635 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:58:33 smithi016 ceph-mon[29214]: from='client.14430 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-18T02:58:33.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:58:33 smithi123 ceph-mon[36358]: from='client.14430 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-18T02:58:34.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:58:34 smithi016 ceph-mon[29214]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:58:34.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:58:34 smithi123 ceph-mon[36358]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:58:34.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:34.847+0000 7f4199705640 1 -- 172.21.15.16:0/3416202679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41941011d0 msgr2=0x7f41941015a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:34.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:34.847+0000 7f4199705640 1 --2- 172.21.15.16:0/3416202679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41941011d0 0x7f41941015a0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f41840048d0 tx=0x7f4184009da0 comp rx=0 tx=0).stop 2024-06-18T02:58:34.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:34.847+0000 7f4199705640 1 -- 172.21.15.16:0/3416202679 shutdown_connections 2024-06-18T02:58:34.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:34.847+0000 7f4199705640 1 --2- 172.21.15.16:0/3416202679 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4194101b70 0x7f41940fb200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:34.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:34.847+0000 7f4199705640 1 --2- 172.21.15.16:0/3416202679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41941011d0 0x7f41941015a0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:34.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:34.847+0000 7f4199705640 1 -- 172.21.15.16:0/3416202679 >> 172.21.15.16:0/3416202679 conn(0x7f41940f6dc0 msgr2=0x7f41940f91d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:58:34.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:34.847+0000 7f4199705640 1 -- 172.21.15.16:0/3416202679 shutdown_connections 2024-06-18T02:58:34.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:34.847+0000 7f4199705640 1 -- 172.21.15.16:0/3416202679 wait complete. 2024-06-18T02:58:34.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:34.848+0000 7f4199705640 1 Processor -- start 2024-06-18T02:58:34.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:34.848+0000 7f4199705640 1 -- start start 2024-06-18T02:58:34.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:34.848+0000 7f4199705640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41941011d0 0x7f419410dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:58:34.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:34.848+0000 7f4199705640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4194101b70 0x7f419410e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:58:34.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:34.848+0000 7f4199705640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f419410eb50 con 0x7f4194101b70 2024-06-18T02:58:34.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:34.848+0000 7f4199705640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f419410ecc0 con 0x7f41941011d0 2024-06-18T02:58:34.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:34.848+0000 7f41927fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4194101b70 0x7f419410e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:58:34.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:34.848+0000 7f41927fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4194101b70 0x7f419410e500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48408/0 (socket says 172.21.15.16:48408) 2024-06-18T02:58:34.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:34.848+0000 7f41927fc640 1 -- 172.21.15.16:0/725150834 learned_addr learned my addr 172.21.15.16:0/725150834 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:58:34.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:34.848+0000 7f4192ffd640 1 --2- 172.21.15.16:0/725150834 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41941011d0 0x7f419410dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:58:34.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:34.848+0000 7f41927fc640 1 -- 172.21.15.16:0/725150834 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41941011d0 msgr2=0x7f419410dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:34.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:34.848+0000 7f41927fc640 1 --2- 172.21.15.16:0/725150834 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41941011d0 0x7f419410dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:34.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:34.848+0000 7f41927fc640 1 -- 172.21.15.16:0/725150834 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4184005040 con 0x7f4194101b70 2024-06-18T02:58:34.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:34.848+0000 7f4192ffd640 1 --2- 172.21.15.16:0/725150834 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41941011d0 0x7f419410dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-18T02:58:34.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:34.848+0000 7f41927fc640 1 --2- 172.21.15.16:0/725150834 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4194101b70 0x7f419410e500 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f418800c830 tx=0x7f4188004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:58:34.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:34.849+0000 7f417b7fe640 1 -- 172.21.15.16:0/725150834 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f418800a850 con 0x7f4194101b70 2024-06-18T02:58:34.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:34.849+0000 7f4199705640 1 -- 172.21.15.16:0/725150834 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4194117030 con 0x7f4194101b70 2024-06-18T02:58:34.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:34.849+0000 7f417b7fe640 1 -- 172.21.15.16:0/725150834 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f418800ae80 con 0x7f4194101b70 2024-06-18T02:58:34.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:34.849+0000 7f417b7fe640 1 -- 172.21.15.16:0/725150834 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4188012ea0 con 0x7f4194101b70 2024-06-18T02:58:34.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:34.849+0000 7f4199705640 1 -- 172.21.15.16:0/725150834 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4194117540 con 0x7f4194101b70 2024-06-18T02:58:34.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:34.850+0000 7f4199705640 1 -- 172.21.15.16:0/725150834 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41940fc940 con 0x7f4194101b70 2024-06-18T02:58:34.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:34.850+0000 7f417b7fe640 1 -- 172.21.15.16:0/725150834 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f4188012400 con 0x7f4194101b70 2024-06-18T02:58:34.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:34.850+0000 7f417b7fe640 1 --2- 172.21.15.16:0/725150834 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f417c065430 0x7f417c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:58:34.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:34.851+0000 7f417b7fe640 1 -- 172.21.15.16:0/725150834 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f4188082570 con 0x7f4194101b70 2024-06-18T02:58:34.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:34.851+0000 7f4192ffd640 1 --2- 172.21.15.16:0/725150834 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f417c065430 0x7f417c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:58:34.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:34.851+0000 7f4192ffd640 1 --2- 172.21.15.16:0/725150834 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f417c065430 0x7f417c0678e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4184009f80 tx=0x7f4184035040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:58:34.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:34.855+0000 7f417b7fe640 1 -- 172.21.15.16:0/725150834 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f4188050c00 con 0x7f4194101b70 2024-06-18T02:58:35.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:35.061+0000 7f4199705640 1 -- 172.21.15.16:0/725150834 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4194108340 con 0x7f4194101b70 2024-06-18T02:58:35.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:35.062+0000 7f417b7fe640 1 -- 172.21.15.16:0/725150834 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f41880126b0 con 0x7f4194101b70 2024-06-18T02:58:35.062 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:58:35.063 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-18T02:58:35.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:35.063+0000 7f4199705640 1 -- 172.21.15.16:0/725150834 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f417c065430 msgr2=0x7f417c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:35.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:35.063+0000 7f4199705640 1 --2- 172.21.15.16:0/725150834 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f417c065430 0x7f417c0678e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4184009f80 tx=0x7f4184035040 comp rx=0 tx=0).stop 2024-06-18T02:58:35.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:35.063+0000 7f4199705640 1 -- 172.21.15.16:0/725150834 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4194101b70 msgr2=0x7f419410e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:35.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:35.063+0000 7f4199705640 1 --2- 172.21.15.16:0/725150834 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4194101b70 0x7f419410e500 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f418800c830 tx=0x7f4188004430 comp rx=0 tx=0).stop 2024-06-18T02:58:35.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:35.064+0000 7f4199705640 1 -- 172.21.15.16:0/725150834 shutdown_connections 2024-06-18T02:58:35.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:35.064+0000 7f4199705640 1 --2- 172.21.15.16:0/725150834 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f417c065430 0x7f417c0678e0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:35.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:35.064+0000 7f4199705640 1 --2- 172.21.15.16:0/725150834 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4194101b70 0x7f419410e500 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:35.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:35.064+0000 7f4199705640 1 --2- 172.21.15.16:0/725150834 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f41941011d0 0x7f419410dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:35.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:35.064+0000 7f4199705640 1 -- 172.21.15.16:0/725150834 >> 172.21.15.16:0/725150834 conn(0x7f41940f6dc0 msgr2=0x7f41940f7d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:58:35.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:35.064+0000 7f4199705640 1 -- 172.21.15.16:0/725150834 shutdown_connections 2024-06-18T02:58:35.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:35.064+0000 7f4199705640 1 -- 172.21.15.16:0/725150834 wait complete. 2024-06-18T02:58:35.688 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:58:35 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/725150834' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:58:35.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:58:35 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/725150834' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:58:36.699 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:58:36 smithi123 ceph-mon[36358]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:58:36.699 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:58:36 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:58:36.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:58:36 smithi016 ceph-mon[29214]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:58:36.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:58:36 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:58:37.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:58:37 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:58:37.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:58:37 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T02:58:37.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:58:37 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:58:37.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:58:37 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:58:37.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:58:37 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T02:58:37.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:58:37 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:58:38.636 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:58:38 smithi123 ceph-mon[36358]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:58:38.718 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T02:58:38.747 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:58:38 smithi016 ceph-mon[29214]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:58:38.994 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T02:58:40.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:40.543+0000 7f1fea3ad640 1 -- 172.21.15.16:0/999506463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fe4102c60 msgr2=0x7f1fe40fc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:40.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:40.543+0000 7f1fea3ad640 1 --2- 172.21.15.16:0/999506463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fe4102c60 0x7f1fe40fc2f0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f1fcc0048d0 tx=0x7f1fcc009da0 comp rx=0 tx=0).stop 2024-06-18T02:58:40.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:40.543+0000 7f1fea3ad640 1 -- 172.21.15.16:0/999506463 shutdown_connections 2024-06-18T02:58:40.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:40.543+0000 7f1fea3ad640 1 --2- 172.21.15.16:0/999506463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fe4102c60 0x7f1fe40fc2f0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:40.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:40.543+0000 7f1fea3ad640 1 --2- 172.21.15.16:0/999506463 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1fe41022c0 0x7f1fe4102690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:40.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:40.543+0000 7f1fea3ad640 1 -- 172.21.15.16:0/999506463 >> 172.21.15.16:0/999506463 conn(0x7f1fe40f7e50 msgr2=0x7f1fe40fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:58:40.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:40.543+0000 7f1fea3ad640 1 -- 172.21.15.16:0/999506463 shutdown_connections 2024-06-18T02:58:40.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:40.543+0000 7f1fea3ad640 1 -- 172.21.15.16:0/999506463 wait complete. 2024-06-18T02:58:40.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:40.544+0000 7f1fea3ad640 1 Processor -- start 2024-06-18T02:58:40.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:40.544+0000 7f1fea3ad640 1 -- start start 2024-06-18T02:58:40.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:40.544+0000 7f1fea3ad640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fe41022c0 0x7f1fe4113370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:58:40.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:40.544+0000 7f1fea3ad640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1fe4102c60 0x7f1fe41138b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:58:40.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:40.544+0000 7f1fea3ad640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe4113eb0 con 0x7f1fe41022c0 2024-06-18T02:58:40.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:40.544+0000 7f1fea3ad640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe410d2f0 con 0x7f1fe4102c60 2024-06-18T02:58:40.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:40.544+0000 7f1fe3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fe41022c0 0x7f1fe4113370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:58:40.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:40.544+0000 7f1fe3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fe41022c0 0x7f1fe4113370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48418/0 (socket says 172.21.15.16:48418) 2024-06-18T02:58:40.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:40.544+0000 7f1fe3fff640 1 -- 172.21.15.16:0/929127158 learned_addr learned my addr 172.21.15.16:0/929127158 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:58:40.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:40.544+0000 7f1fdbfff640 1 --2- 172.21.15.16:0/929127158 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1fe4102c60 0x7f1fe41138b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:58:40.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:40.544+0000 7f1fe3fff640 1 -- 172.21.15.16:0/929127158 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1fe4102c60 msgr2=0x7f1fe41138b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:40.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:40.544+0000 7f1fe3fff640 1 --2- 172.21.15.16:0/929127158 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1fe4102c60 0x7f1fe41138b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:40.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:40.544+0000 7f1fe3fff640 1 -- 172.21.15.16:0/929127158 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fcc005040 con 0x7f1fe41022c0 2024-06-18T02:58:40.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:40.545+0000 7f1fe3fff640 1 --2- 172.21.15.16:0/929127158 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fe41022c0 0x7f1fe4113370 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f1fd4002a50 tx=0x7f1fd4004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:58:40.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:40.545+0000 7f1fe17fa640 1 -- 172.21.15.16:0/929127158 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1fd4007c30 con 0x7f1fe41022c0 2024-06-18T02:58:40.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:40.545+0000 7f1fe17fa640 1 -- 172.21.15.16:0/929127158 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1fd400f040 con 0x7f1fe41022c0 2024-06-18T02:58:40.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:40.545+0000 7f1fea3ad640 1 -- 172.21.15.16:0/929127158 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fe410d5d0 con 0x7f1fe41022c0 2024-06-18T02:58:40.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:40.545+0000 7f1fe17fa640 1 -- 172.21.15.16:0/929127158 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1fd400dad0 con 0x7f1fe41022c0 2024-06-18T02:58:40.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:40.545+0000 7f1fea3ad640 1 -- 172.21.15.16:0/929127158 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fe410db40 con 0x7f1fe41022c0 2024-06-18T02:58:40.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:40.546+0000 7f1fea3ad640 1 -- 172.21.15.16:0/929127158 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fe40fda30 con 0x7f1fe41022c0 2024-06-18T02:58:40.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:40.546+0000 7f1fe17fa640 1 -- 172.21.15.16:0/929127158 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f1fd400dc70 con 0x7f1fe41022c0 2024-06-18T02:58:40.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:40.547+0000 7f1fe17fa640 1 --2- 172.21.15.16:0/929127158 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f1fd0065430 0x7f1fd00678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:58:40.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:40.547+0000 7f1fe17fa640 1 -- 172.21.15.16:0/929127158 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f1fd4081bb0 con 0x7f1fe41022c0 2024-06-18T02:58:40.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:40.547+0000 7f1fdbfff640 1 --2- 172.21.15.16:0/929127158 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f1fd0065430 0x7f1fd00678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:58:40.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:40.547+0000 7f1fdbfff640 1 --2- 172.21.15.16:0/929127158 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f1fd0065430 0x7f1fd00678e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1fcc009f80 tx=0x7f1fcc035040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:58:40.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:40.551+0000 7f1fe17fa640 1 -- 172.21.15.16:0/929127158 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f1fd404cdf0 con 0x7f1fe41022c0 2024-06-18T02:58:40.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:58:40 smithi123 ceph-mon[36358]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:58:40.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:40.760+0000 7f1fea3ad640 1 -- 172.21.15.16:0/929127158 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f1fe4109420 con 0x7f1fe41022c0 2024-06-18T02:58:40.762 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:58:40 smithi016 ceph-mon[29214]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:58:40.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:40.760+0000 7f1fe17fa640 1 -- 172.21.15.16:0/929127158 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f1fd4088080 con 0x7f1fe41022c0 2024-06-18T02:58:40.762 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:58:40.762 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-18T02:58:40.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:40.763+0000 7f1fea3ad640 1 -- 172.21.15.16:0/929127158 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f1fd0065430 msgr2=0x7f1fd00678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:40.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:40.763+0000 7f1fea3ad640 1 --2- 172.21.15.16:0/929127158 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f1fd0065430 0x7f1fd00678e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1fcc009f80 tx=0x7f1fcc035040 comp rx=0 tx=0).stop 2024-06-18T02:58:40.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:40.763+0000 7f1fea3ad640 1 -- 172.21.15.16:0/929127158 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fe41022c0 msgr2=0x7f1fe4113370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:40.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:40.763+0000 7f1fea3ad640 1 --2- 172.21.15.16:0/929127158 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fe41022c0 0x7f1fe4113370 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f1fd4002a50 tx=0x7f1fd4004600 comp rx=0 tx=0).stop 2024-06-18T02:58:40.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:40.763+0000 7f1fea3ad640 1 -- 172.21.15.16:0/929127158 shutdown_connections 2024-06-18T02:58:40.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:40.763+0000 7f1fea3ad640 1 --2- 172.21.15.16:0/929127158 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f1fd0065430 0x7f1fd00678e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:40.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:40.763+0000 7f1fea3ad640 1 --2- 172.21.15.16:0/929127158 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1fe4102c60 0x7f1fe41138b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:40.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:40.763+0000 7f1fea3ad640 1 --2- 172.21.15.16:0/929127158 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fe41022c0 0x7f1fe4113370 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:40.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:40.763+0000 7f1fea3ad640 1 -- 172.21.15.16:0/929127158 >> 172.21.15.16:0/929127158 conn(0x7f1fe40f7e50 msgr2=0x7f1fe40f8db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:58:40.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:40.763+0000 7f1fea3ad640 1 -- 172.21.15.16:0/929127158 shutdown_connections 2024-06-18T02:58:40.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:40.763+0000 7f1fea3ad640 1 -- 172.21.15.16:0/929127158 wait complete. 2024-06-18T02:58:41.510 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:58:41 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/929127158' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:58:41.511 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:58:41 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:58:41.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:58:41 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/929127158' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:58:41.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:58:41 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:58:42.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:58:42 smithi123 ceph-mon[36358]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:58:42.785 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:58:42 smithi016 ceph-mon[29214]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:58:44.541 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T02:58:44.554 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:58:44 smithi016 ceph-mon[29214]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:58:44.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:58:44 smithi123 ceph-mon[36358]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:58:44.819 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T02:58:46.538 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:58:46 smithi016 ceph-mon[29214]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:58:46.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:46.516+0000 7fbcb3d4c640 1 -- 172.21.15.16:0/1395285433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbcac071440 msgr2=0x7fbcac06f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:46.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:46.516+0000 7fbcb3d4c640 1 --2- 172.21.15.16:0/1395285433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbcac071440 0x7fbcac06f810 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fbca80048d0 tx=0x7fbca8009da0 comp rx=0 tx=0).stop 2024-06-18T02:58:46.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:46.516+0000 7fbcb3d4c640 1 -- 172.21.15.16:0/1395285433 shutdown_connections 2024-06-18T02:58:46.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:46.516+0000 7fbcb3d4c640 1 --2- 172.21.15.16:0/1395285433 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbcac06fd50 0x7fbcac0701c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:46.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:46.516+0000 7fbcb3d4c640 1 --2- 172.21.15.16:0/1395285433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbcac071440 0x7fbcac06f810 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:46.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:46.516+0000 7fbcb3d4c640 1 -- 172.21.15.16:0/1395285433 >> 172.21.15.16:0/1395285433 conn(0x7fbcac0f7f00 msgr2=0x7fbcac0fa310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:58:46.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:46.516+0000 7fbcb3d4c640 1 -- 172.21.15.16:0/1395285433 shutdown_connections 2024-06-18T02:58:46.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:46.516+0000 7fbcb3d4c640 1 -- 172.21.15.16:0/1395285433 wait complete. 2024-06-18T02:58:46.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:46.517+0000 7fbcb3d4c640 1 Processor -- start 2024-06-18T02:58:46.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:46.517+0000 7fbcb3d4c640 1 -- start start 2024-06-18T02:58:46.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:46.517+0000 7fbcb3d4c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbcac06fd50 0x7fbcac119690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:58:46.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:46.517+0000 7fbcb3d4c640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbcac071440 0x7fbcac119bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:58:46.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:46.517+0000 7fbcb3d4c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcac11a240 con 0x7fbcac06fd50 2024-06-18T02:58:46.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:46.517+0000 7fbcb3d4c640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcac113630 con 0x7fbcac071440 2024-06-18T02:58:46.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:46.517+0000 7fbcb1ae8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbcac06fd50 0x7fbcac119690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:58:46.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:46.517+0000 7fbcb1ae8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbcac06fd50 0x7fbcac119690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48060/0 (socket says 172.21.15.16:48060) 2024-06-18T02:58:46.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:46.517+0000 7fbcb1ae8640 1 -- 172.21.15.16:0/3960696234 learned_addr learned my addr 172.21.15.16:0/3960696234 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:58:46.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:46.517+0000 7fbcb12e7640 1 --2- 172.21.15.16:0/3960696234 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbcac071440 0x7fbcac119bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:58:46.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:46.517+0000 7fbcb1ae8640 1 -- 172.21.15.16:0/3960696234 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbcac071440 msgr2=0x7fbcac119bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:46.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:46.517+0000 7fbcb1ae8640 1 --2- 172.21.15.16:0/3960696234 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbcac071440 0x7fbcac119bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:46.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:46.517+0000 7fbcb1ae8640 1 -- 172.21.15.16:0/3960696234 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbca8005040 con 0x7fbcac06fd50 2024-06-18T02:58:46.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:46.518+0000 7fbcb1ae8640 1 --2- 172.21.15.16:0/3960696234 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbcac06fd50 0x7fbcac119690 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fbca80043d0 tx=0x7fbca8009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:58:46.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:46.518+0000 7fbca27fc640 1 -- 172.21.15.16:0/3960696234 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbca8038070 con 0x7fbcac06fd50 2024-06-18T02:58:46.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:46.518+0000 7fbca27fc640 1 -- 172.21.15.16:0/3960696234 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbca8032d30 con 0x7fbcac06fd50 2024-06-18T02:58:46.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:46.518+0000 7fbcb3d4c640 1 -- 172.21.15.16:0/3960696234 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcac1138b0 con 0x7fbcac06fd50 2024-06-18T02:58:46.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:46.518+0000 7fbca27fc640 1 -- 172.21.15.16:0/3960696234 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbca803dcd0 con 0x7fbcac06fd50 2024-06-18T02:58:46.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:46.518+0000 7fbcb3d4c640 1 -- 172.21.15.16:0/3960696234 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcac113d70 con 0x7fbcac06fd50 2024-06-18T02:58:46.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:46.519+0000 7fbcb3d4c640 1 -- 172.21.15.16:0/3960696234 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcac070f50 con 0x7fbcac06fd50 2024-06-18T02:58:46.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:46.519+0000 7fbca27fc640 1 -- 172.21.15.16:0/3960696234 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fbca8030020 con 0x7fbcac06fd50 2024-06-18T02:58:46.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:46.520+0000 7fbca27fc640 1 --2- 172.21.15.16:0/3960696234 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fbc98065430 0x7fbc980678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:58:46.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:46.520+0000 7fbca27fc640 1 -- 172.21.15.16:0/3960696234 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fbca80a2fa0 con 0x7fbcac06fd50 2024-06-18T02:58:46.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:46.520+0000 7fbcb12e7640 1 --2- 172.21.15.16:0/3960696234 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fbc98065430 0x7fbc980678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:58:46.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:46.520+0000 7fbcb12e7640 1 --2- 172.21.15.16:0/3960696234 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fbc98065430 0x7fbc980678e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fbcac114f50 tx=0x7fbc9c00b040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:58:46.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:46.524+0000 7fbca27fc640 1 -- 172.21.15.16:0/3960696234 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fbca80714d0 con 0x7fbcac06fd50 2024-06-18T02:58:46.699 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:58:46 smithi123 ceph-mon[36358]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:58:46.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:46.738+0000 7fbcb3d4c640 1 -- 172.21.15.16:0/3960696234 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fbcac04b480 con 0x7fbcac06fd50 2024-06-18T02:58:46.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:46.739+0000 7fbca27fc640 1 -- 172.21.15.16:0/3960696234 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fbca8074af0 con 0x7fbcac06fd50 2024-06-18T02:58:46.739 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:58:46.739 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-18T02:58:46.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:46.741+0000 7fbcb3d4c640 1 -- 172.21.15.16:0/3960696234 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fbc98065430 msgr2=0x7fbc980678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:46.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:46.741+0000 7fbcb3d4c640 1 --2- 172.21.15.16:0/3960696234 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fbc98065430 0x7fbc980678e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fbcac114f50 tx=0x7fbc9c00b040 comp rx=0 tx=0).stop 2024-06-18T02:58:46.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:46.741+0000 7fbcb3d4c640 1 -- 172.21.15.16:0/3960696234 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbcac06fd50 msgr2=0x7fbcac119690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:46.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:46.741+0000 7fbcb3d4c640 1 --2- 172.21.15.16:0/3960696234 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbcac06fd50 0x7fbcac119690 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fbca80043d0 tx=0x7fbca8009da0 comp rx=0 tx=0).stop 2024-06-18T02:58:46.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:46.741+0000 7fbcb3d4c640 1 -- 172.21.15.16:0/3960696234 shutdown_connections 2024-06-18T02:58:46.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:46.741+0000 7fbcb3d4c640 1 --2- 172.21.15.16:0/3960696234 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fbc98065430 0x7fbc980678e0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:46.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:46.741+0000 7fbcb3d4c640 1 --2- 172.21.15.16:0/3960696234 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbcac071440 0x7fbcac119bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:46.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:46.741+0000 7fbcb3d4c640 1 --2- 172.21.15.16:0/3960696234 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbcac06fd50 0x7fbcac119690 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:46.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:46.741+0000 7fbcb3d4c640 1 -- 172.21.15.16:0/3960696234 >> 172.21.15.16:0/3960696234 conn(0x7fbcac0f7f00 msgr2=0x7fbcac0f9080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:58:46.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:46.741+0000 7fbcb3d4c640 1 -- 172.21.15.16:0/3960696234 shutdown_connections 2024-06-18T02:58:46.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:46.741+0000 7fbcb3d4c640 1 -- 172.21.15.16:0/3960696234 wait complete. 2024-06-18T02:58:47.483 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:58:47 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/3960696234' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:58:47.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:58:47 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/3960696234' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:58:48.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:58:48 smithi123 ceph-mon[36358]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:58:48.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:58:48 smithi016 ceph-mon[29214]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:58:50.515 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T02:58:50.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:58:50 smithi123 ceph-mon[36358]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:58:50.791 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T02:58:50.804 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:58:50 smithi016 ceph-mon[29214]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:58:52.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:52.381+0000 7f07581b8640 1 -- 172.21.15.16:0/3453084996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07500fbe80 msgr2=0x7f07500fcd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:52.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:52.381+0000 7f07581b8640 1 --2- 172.21.15.16:0/3453084996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07500fbe80 0x7f07500fcd80 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f07440048d0 tx=0x7f0744009da0 comp rx=0 tx=0).stop 2024-06-18T02:58:52.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:52.381+0000 7f07581b8640 1 -- 172.21.15.16:0/3453084996 shutdown_connections 2024-06-18T02:58:52.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:52.381+0000 7f07581b8640 1 --2- 172.21.15.16:0/3453084996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07500fbe80 0x7f07500fcd80 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:52.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:52.381+0000 7f07581b8640 1 --2- 172.21.15.16:0/3453084996 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f07500fb570 0x7f07500fb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:52.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:52.381+0000 7f07581b8640 1 -- 172.21.15.16:0/3453084996 >> 172.21.15.16:0/3453084996 conn(0x7f0750072110 msgr2=0x7f0750072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:58:52.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:52.381+0000 7f07581b8640 1 -- 172.21.15.16:0/3453084996 shutdown_connections 2024-06-18T02:58:52.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:52.381+0000 7f07581b8640 1 -- 172.21.15.16:0/3453084996 wait complete. 2024-06-18T02:58:52.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:52.381+0000 7f07581b8640 1 Processor -- start 2024-06-18T02:58:52.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:52.381+0000 7f07581b8640 1 -- start start 2024-06-18T02:58:52.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:52.382+0000 7f07581b8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07500fb570 0x7f0750113040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:58:52.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:52.382+0000 7f07581b8640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f07500fbe80 0x7f0750113580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:58:52.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:52.382+0000 7f07581b8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0750113c10 con 0x7f07500fb570 2024-06-18T02:58:52.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:52.382+0000 7f07581b8640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f075010cfc0 con 0x7f07500fbe80 2024-06-18T02:58:52.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:52.382+0000 7f0755f54640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07500fb570 0x7f0750113040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:58:52.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:52.382+0000 7f0755f54640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07500fb570 0x7f0750113040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36788/0 (socket says 172.21.15.16:36788) 2024-06-18T02:58:52.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:52.382+0000 7f0755f54640 1 -- 172.21.15.16:0/2445899570 learned_addr learned my addr 172.21.15.16:0/2445899570 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:58:52.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:52.382+0000 7f0755f54640 1 -- 172.21.15.16:0/2445899570 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f07500fbe80 msgr2=0x7f0750113580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:52.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:52.382+0000 7f0755f54640 1 --2- 172.21.15.16:0/2445899570 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f07500fbe80 0x7f0750113580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:52.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:52.382+0000 7f0755f54640 1 -- 172.21.15.16:0/2445899570 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0744005040 con 0x7f07500fb570 2024-06-18T02:58:52.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:52.382+0000 7f0755f54640 1 --2- 172.21.15.16:0/2445899570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07500fb570 0x7f0750113040 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f074000bbb0 tx=0x7f0740009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:58:52.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:52.382+0000 7f074e7fc640 1 -- 172.21.15.16:0/2445899570 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f074000c9c0 con 0x7f07500fb570 2024-06-18T02:58:52.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:52.382+0000 7f074e7fc640 1 -- 172.21.15.16:0/2445899570 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f07400092e0 con 0x7f07500fb570 2024-06-18T02:58:52.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:52.382+0000 7f07581b8640 1 -- 172.21.15.16:0/2445899570 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f075010d2a0 con 0x7f07500fb570 2024-06-18T02:58:52.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:52.382+0000 7f074e7fc640 1 -- 172.21.15.16:0/2445899570 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0740007640 con 0x7f07500fb570 2024-06-18T02:58:52.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:52.382+0000 7f07581b8640 1 -- 172.21.15.16:0/2445899570 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f075010d7c0 con 0x7f07500fb570 2024-06-18T02:58:52.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:52.384+0000 7f07581b8640 1 -- 172.21.15.16:0/2445899570 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07500fc920 con 0x7f07500fb570 2024-06-18T02:58:52.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:52.384+0000 7f074e7fc640 1 -- 172.21.15.16:0/2445899570 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f0740012070 con 0x7f07500fb570 2024-06-18T02:58:52.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:52.384+0000 7f074e7fc640 1 --2- 172.21.15.16:0/2445899570 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f0738065430 0x7f07380678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:58:52.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:52.384+0000 7f074e7fc640 1 -- 172.21.15.16:0/2445899570 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f0740082620 con 0x7f07500fb570 2024-06-18T02:58:52.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:52.384+0000 7f0755753640 1 --2- 172.21.15.16:0/2445899570 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f0738065430 0x7f07380678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:58:52.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:52.385+0000 7f0755753640 1 --2- 172.21.15.16:0/2445899570 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f0738065430 0x7f07380678e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f074402a530 tx=0x7f0744009da0 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:58:52.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:52.388+0000 7f074e7fc640 1 -- 172.21.15.16:0/2445899570 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f0740050a30 con 0x7f07500fb570 2024-06-18T02:58:52.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:58:52 smithi016 ceph-mon[29214]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:58:52.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:52.598+0000 7f07581b8640 1 -- 172.21.15.16:0/2445899570 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f075010e710 con 0x7f07500fb570 2024-06-18T02:58:52.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:52.598+0000 7f074e7fc640 1 -- 172.21.15.16:0/2445899570 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f0740054050 con 0x7f07500fb570 2024-06-18T02:58:52.599 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:58:52.599 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-18T02:58:52.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:52.600+0000 7f07581b8640 1 -- 172.21.15.16:0/2445899570 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f0738065430 msgr2=0x7f07380678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:52.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:52.600+0000 7f07581b8640 1 --2- 172.21.15.16:0/2445899570 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f0738065430 0x7f07380678e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f074402a530 tx=0x7f0744009da0 comp rx=0 tx=0).stop 2024-06-18T02:58:52.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:52.600+0000 7f07581b8640 1 -- 172.21.15.16:0/2445899570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07500fb570 msgr2=0x7f0750113040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:52.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:52.600+0000 7f07581b8640 1 --2- 172.21.15.16:0/2445899570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07500fb570 0x7f0750113040 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f074000bbb0 tx=0x7f0740009840 comp rx=0 tx=0).stop 2024-06-18T02:58:52.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:52.600+0000 7f07581b8640 1 -- 172.21.15.16:0/2445899570 shutdown_connections 2024-06-18T02:58:52.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:52.600+0000 7f07581b8640 1 --2- 172.21.15.16:0/2445899570 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f0738065430 0x7f07380678e0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:52.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:52.600+0000 7f07581b8640 1 --2- 172.21.15.16:0/2445899570 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f07500fbe80 0x7f0750113580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:52.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:52.600+0000 7f07581b8640 1 --2- 172.21.15.16:0/2445899570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07500fb570 0x7f0750113040 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:52.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:52.600+0000 7f07581b8640 1 -- 172.21.15.16:0/2445899570 >> 172.21.15.16:0/2445899570 conn(0x7f0750072110 msgr2=0x7f07500ffea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:58:52.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:52.600+0000 7f07581b8640 1 -- 172.21.15.16:0/2445899570 shutdown_connections 2024-06-18T02:58:52.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:52.600+0000 7f07581b8640 1 -- 172.21.15.16:0/2445899570 wait complete. 2024-06-18T02:58:52.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:58:52 smithi123 ceph-mon[36358]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:58:53.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:58:53 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/2445899570' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:58:53.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:58:53 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/2445899570' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:58:54.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:58:54 smithi016 ceph-mon[29214]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:58:54.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:58:54 smithi123 ceph-mon[36358]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:58:56.233 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T02:58:56.515 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T02:58:56.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:58:56 smithi016 ceph-mon[29214]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:58:56.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:58:56 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:58:56.699 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:58:56 smithi123 ceph-mon[36358]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:58:56.699 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:58:56 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:58:58.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:58.120+0000 7f070f99d640 1 -- 172.21.15.16:0/2051634465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0708102c60 msgr2=0x7f07080fc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:58.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:58.120+0000 7f070f99d640 1 --2- 172.21.15.16:0/2051634465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0708102c60 0x7f07080fc2f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f06f80048d0 tx=0x7f06f8009da0 comp rx=0 tx=0).stop 2024-06-18T02:58:58.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:58.120+0000 7f070f99d640 1 -- 172.21.15.16:0/2051634465 shutdown_connections 2024-06-18T02:58:58.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:58.120+0000 7f070f99d640 1 --2- 172.21.15.16:0/2051634465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0708102c60 0x7f07080fc2f0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:58.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:58.120+0000 7f070f99d640 1 --2- 172.21.15.16:0/2051634465 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f07081022c0 0x7f0708102690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:58.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:58.120+0000 7f070f99d640 1 -- 172.21.15.16:0/2051634465 >> 172.21.15.16:0/2051634465 conn(0x7f07080f7e50 msgr2=0x7f07080fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:58:58.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:58.120+0000 7f070f99d640 1 -- 172.21.15.16:0/2051634465 shutdown_connections 2024-06-18T02:58:58.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:58.120+0000 7f070f99d640 1 -- 172.21.15.16:0/2051634465 wait complete. 2024-06-18T02:58:58.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:58.121+0000 7f070f99d640 1 Processor -- start 2024-06-18T02:58:58.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:58.121+0000 7f070f99d640 1 -- start start 2024-06-18T02:58:58.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:58.121+0000 7f070f99d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07081022c0 0x7f0708113350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:58:58.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:58.121+0000 7f070f99d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0708102c60 0x7f0708113890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:58:58.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:58.121+0000 7f070f99d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0708113e90 con 0x7f07081022c0 2024-06-18T02:58:58.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:58.121+0000 7f070f99d640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f070810d2d0 con 0x7f0708102c60 2024-06-18T02:58:58.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:58.121+0000 7f070d739640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07081022c0 0x7f0708113350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:58:58.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:58.121+0000 7f070d739640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07081022c0 0x7f0708113350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36818/0 (socket says 172.21.15.16:36818) 2024-06-18T02:58:58.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:58.121+0000 7f070d739640 1 -- 172.21.15.16:0/1213063155 learned_addr learned my addr 172.21.15.16:0/1213063155 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:58:58.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:58.122+0000 7f070cf38640 1 --2- 172.21.15.16:0/1213063155 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0708102c60 0x7f0708113890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:58:58.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:58.122+0000 7f070d739640 1 -- 172.21.15.16:0/1213063155 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0708102c60 msgr2=0x7f0708113890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:58.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:58.122+0000 7f070d739640 1 --2- 172.21.15.16:0/1213063155 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0708102c60 0x7f0708113890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:58.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:58.122+0000 7f070d739640 1 -- 172.21.15.16:0/1213063155 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06f8005040 con 0x7f07081022c0 2024-06-18T02:58:58.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:58.122+0000 7f070d739640 1 --2- 172.21.15.16:0/1213063155 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07081022c0 0x7f0708113350 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f070400c830 tx=0x7f0704004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:58:58.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:58.122+0000 7f06fdffb640 1 -- 172.21.15.16:0/1213063155 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f070400a7d0 con 0x7f07081022c0 2024-06-18T02:58:58.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:58.122+0000 7f06fdffb640 1 -- 172.21.15.16:0/1213063155 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f070400ae00 con 0x7f07081022c0 2024-06-18T02:58:58.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:58.122+0000 7f070f99d640 1 -- 172.21.15.16:0/1213063155 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f070810d5b0 con 0x7f07081022c0 2024-06-18T02:58:58.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:58.122+0000 7f06fdffb640 1 -- 172.21.15.16:0/1213063155 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f07040113a0 con 0x7f07081022c0 2024-06-18T02:58:58.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:58.122+0000 7f070f99d640 1 -- 172.21.15.16:0/1213063155 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f070810db20 con 0x7f07081022c0 2024-06-18T02:58:58.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:58.123+0000 7f070f99d640 1 -- 172.21.15.16:0/1213063155 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07080fda30 con 0x7f07081022c0 2024-06-18T02:58:58.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:58.124+0000 7f06fdffb640 1 -- 172.21.15.16:0/1213063155 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f0704012400 con 0x7f07081022c0 2024-06-18T02:58:58.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:58.124+0000 7f06fdffb640 1 --2- 172.21.15.16:0/1213063155 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f06f4065430 0x7f06f40678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:58:58.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:58.124+0000 7f06fdffb640 1 -- 172.21.15.16:0/1213063155 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f07040825e0 con 0x7f07081022c0 2024-06-18T02:58:58.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:58.124+0000 7f070cf38640 1 --2- 172.21.15.16:0/1213063155 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f06f4065430 0x7f06f40678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:58:58.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:58.125+0000 7f070cf38640 1 --2- 172.21.15.16:0/1213063155 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f06f4065430 0x7f06f40678e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f06f8009f80 tx=0x7f06f8035040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:58:58.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:58.128+0000 7f06fdffb640 1 -- 172.21.15.16:0/1213063155 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f0704050b10 con 0x7f07081022c0 2024-06-18T02:58:58.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:58.339+0000 7f070f99d640 1 -- 172.21.15.16:0/1213063155 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f07080fbab0 con 0x7f07081022c0 2024-06-18T02:58:58.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:58.339+0000 7f06fdffb640 1 -- 172.21.15.16:0/1213063155 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f0704054130 con 0x7f07081022c0 2024-06-18T02:58:58.340 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:58:58.340 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-18T02:58:58.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:58.341+0000 7f070f99d640 1 -- 172.21.15.16:0/1213063155 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f06f4065430 msgr2=0x7f06f40678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:58.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:58.341+0000 7f070f99d640 1 --2- 172.21.15.16:0/1213063155 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f06f4065430 0x7f06f40678e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f06f8009f80 tx=0x7f06f8035040 comp rx=0 tx=0).stop 2024-06-18T02:58:58.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:58.341+0000 7f070f99d640 1 -- 172.21.15.16:0/1213063155 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07081022c0 msgr2=0x7f0708113350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:58:58.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:58.341+0000 7f070f99d640 1 --2- 172.21.15.16:0/1213063155 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07081022c0 0x7f0708113350 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f070400c830 tx=0x7f0704004430 comp rx=0 tx=0).stop 2024-06-18T02:58:58.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:58.341+0000 7f070f99d640 1 -- 172.21.15.16:0/1213063155 shutdown_connections 2024-06-18T02:58:58.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:58.341+0000 7f070f99d640 1 --2- 172.21.15.16:0/1213063155 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f06f4065430 0x7f06f40678e0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:58.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:58.341+0000 7f070f99d640 1 --2- 172.21.15.16:0/1213063155 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0708102c60 0x7f0708113890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:58.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:58.341+0000 7f070f99d640 1 --2- 172.21.15.16:0/1213063155 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f07081022c0 0x7f0708113350 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:58:58.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:58.341+0000 7f070f99d640 1 -- 172.21.15.16:0/1213063155 >> 172.21.15.16:0/1213063155 conn(0x7f07080f7e50 msgr2=0x7f07080f8db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:58:58.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:58.341+0000 7f070f99d640 1 -- 172.21.15.16:0/1213063155 shutdown_connections 2024-06-18T02:58:58.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:58:58.341+0000 7f070f99d640 1 -- 172.21.15.16:0/1213063155 wait complete. 2024-06-18T02:58:58.693 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:58:58 smithi016 ceph-mon[29214]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:58:58.694 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:58:58 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/1213063155' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:58:58.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:58:58 smithi123 ceph-mon[36358]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:58:58.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:58:58 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/1213063155' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:59:00.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:59:00 smithi123 ceph-mon[36358]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:59:00.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:59:00 smithi016 ceph-mon[29214]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:59:02.013 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T02:59:02.290 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T02:59:02.687 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:59:02 smithi016 ceph-mon[29214]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:59:02.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:59:02 smithi123 ceph-mon[36358]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:59:03.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:03.926+0000 7f3106082640 1 -- 172.21.15.16:0/2525605788 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3100101200 msgr2=0x7f31001015d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:59:03.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:03.926+0000 7f3106082640 1 --2- 172.21.15.16:0/2525605788 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3100101200 0x7f31001015d0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f30f00048d0 tx=0x7f30f0009da0 comp rx=0 tx=0).stop 2024-06-18T02:59:03.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:03.926+0000 7f3106082640 1 -- 172.21.15.16:0/2525605788 shutdown_connections 2024-06-18T02:59:03.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:03.926+0000 7f3106082640 1 --2- 172.21.15.16:0/2525605788 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3100101ba0 0x7f31000fb230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:59:03.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:03.926+0000 7f3106082640 1 --2- 172.21.15.16:0/2525605788 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3100101200 0x7f31001015d0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:59:03.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:03.926+0000 7f3106082640 1 -- 172.21.15.16:0/2525605788 >> 172.21.15.16:0/2525605788 conn(0x7f31000f6df0 msgr2=0x7f31000f9200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:59:03.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:03.926+0000 7f3106082640 1 -- 172.21.15.16:0/2525605788 shutdown_connections 2024-06-18T02:59:03.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:03.926+0000 7f3106082640 1 -- 172.21.15.16:0/2525605788 wait complete. 2024-06-18T02:59:03.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:03.927+0000 7f3106082640 1 Processor -- start 2024-06-18T02:59:03.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:03.927+0000 7f3106082640 1 -- start start 2024-06-18T02:59:03.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:03.927+0000 7f3106082640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3100101200 0x7f3100111ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:59:03.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:03.927+0000 7f3106082640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3100101ba0 0x7f3100112410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:59:03.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:03.927+0000 7f3106082640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3100112a10 con 0x7f3100101200 2024-06-18T02:59:03.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:03.927+0000 7f3106082640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f310010be50 con 0x7f3100101ba0 2024-06-18T02:59:03.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:03.927+0000 7f30ff7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3100101200 0x7f3100111ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:59:03.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:03.927+0000 7f30ff7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3100101200 0x7f3100111ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35470/0 (socket says 172.21.15.16:35470) 2024-06-18T02:59:03.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:03.927+0000 7f30ff7fe640 1 -- 172.21.15.16:0/1815269970 learned_addr learned my addr 172.21.15.16:0/1815269970 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:59:03.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:03.927+0000 7f30feffd640 1 --2- 172.21.15.16:0/1815269970 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3100101ba0 0x7f3100112410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:59:03.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:03.928+0000 7f30feffd640 1 -- 172.21.15.16:0/1815269970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3100101200 msgr2=0x7f3100111ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:59:03.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:03.928+0000 7f30feffd640 1 --2- 172.21.15.16:0/1815269970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3100101200 0x7f3100111ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:59:03.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:03.928+0000 7f30feffd640 1 -- 172.21.15.16:0/1815269970 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30f0005040 con 0x7f3100101ba0 2024-06-18T02:59:03.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:03.928+0000 7f30feffd640 1 --2- 172.21.15.16:0/1815269970 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3100101ba0 0x7f3100112410 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f30f40027d0 tx=0x7f30f4009f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:59:03.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:03.929+0000 7f30e7fff640 1 -- 172.21.15.16:0/1815269970 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f30f4002dc0 con 0x7f3100101ba0 2024-06-18T02:59:03.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:03.929+0000 7f30e7fff640 1 -- 172.21.15.16:0/1815269970 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f30f400f040 con 0x7f3100101ba0 2024-06-18T02:59:03.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:03.929+0000 7f3106082640 1 -- 172.21.15.16:0/1815269970 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f310010c130 con 0x7f3100101ba0 2024-06-18T02:59:03.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:03.929+0000 7f30e7fff640 1 -- 172.21.15.16:0/1815269970 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f30f40095f0 con 0x7f3100101ba0 2024-06-18T02:59:03.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:03.929+0000 7f3106082640 1 -- 172.21.15.16:0/1815269970 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f310010c680 con 0x7f3100101ba0 2024-06-18T02:59:03.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:03.930+0000 7f3106082640 1 -- 172.21.15.16:0/1815269970 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31000fc970 con 0x7f3100101ba0 2024-06-18T02:59:03.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:03.930+0000 7f30e7fff640 1 -- 172.21.15.16:0/1815269970 <== mon.1 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f30f4009750 con 0x7f3100101ba0 2024-06-18T02:59:03.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:03.930+0000 7f30e7fff640 1 --2- 172.21.15.16:0/1815269970 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f30e8065430 0x7f30e80678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:59:03.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:03.930+0000 7f30e7fff640 1 -- 172.21.15.16:0/1815269970 <== mon.1 v2:172.21.15.123:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f30f4081d00 con 0x7f3100101ba0 2024-06-18T02:59:03.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:03.931+0000 7f30ff7fe640 1 --2- 172.21.15.16:0/1815269970 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f30e8065430 0x7f30e80678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:59:03.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:03.931+0000 7f30ff7fe640 1 --2- 172.21.15.16:0/1815269970 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f30e8065430 0x7f30e80678e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f30f0000c00 tx=0x7f30f0035040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:59:03.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:03.935+0000 7f30e7fff640 1 -- 172.21.15.16:0/1815269970 <== mon.1 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f30f404cdf0 con 0x7f3100101ba0 2024-06-18T02:59:04.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:04.143+0000 7f3106082640 1 -- 172.21.15.16:0/1815269970 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f3100108360 con 0x7f3100101ba0 2024-06-18T02:59:04.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:04.144+0000 7f30e7fff640 1 -- 172.21.15.16:0/1815269970 <== mon.1 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f30f40501c0 con 0x7f3100101ba0 2024-06-18T02:59:04.144 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:59:04.145 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-18T02:59:04.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:04.146+0000 7f3106082640 1 -- 172.21.15.16:0/1815269970 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f30e8065430 msgr2=0x7f30e80678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:59:04.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:04.146+0000 7f3106082640 1 --2- 172.21.15.16:0/1815269970 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f30e8065430 0x7f30e80678e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f30f0000c00 tx=0x7f30f0035040 comp rx=0 tx=0).stop 2024-06-18T02:59:04.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:04.146+0000 7f3106082640 1 -- 172.21.15.16:0/1815269970 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3100101ba0 msgr2=0x7f3100112410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:59:04.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:04.146+0000 7f3106082640 1 --2- 172.21.15.16:0/1815269970 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3100101ba0 0x7f3100112410 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f30f40027d0 tx=0x7f30f4009f70 comp rx=0 tx=0).stop 2024-06-18T02:59:04.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:04.146+0000 7f3106082640 1 -- 172.21.15.16:0/1815269970 shutdown_connections 2024-06-18T02:59:04.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:04.146+0000 7f3106082640 1 --2- 172.21.15.16:0/1815269970 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f30e8065430 0x7f30e80678e0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:59:04.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:04.146+0000 7f3106082640 1 --2- 172.21.15.16:0/1815269970 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3100101ba0 0x7f3100112410 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:59:04.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:04.146+0000 7f3106082640 1 --2- 172.21.15.16:0/1815269970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3100101200 0x7f3100111ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:59:04.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:04.146+0000 7f3106082640 1 -- 172.21.15.16:0/1815269970 >> 172.21.15.16:0/1815269970 conn(0x7f31000f6df0 msgr2=0x7f31000f7db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:59:04.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:04.146+0000 7f3106082640 1 -- 172.21.15.16:0/1815269970 shutdown_connections 2024-06-18T02:59:04.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:04.146+0000 7f3106082640 1 -- 172.21.15.16:0/1815269970 wait complete. 2024-06-18T02:59:04.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:59:04 smithi016 ceph-mon[29214]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:59:04.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:59:04 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/1815269970' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:59:04.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:59:04 smithi123 ceph-mon[36358]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:59:04.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:59:04 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/1815269970' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:59:06.700 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:59:06 smithi123 ceph-mon[36358]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:59:06.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:59:06 smithi016 ceph-mon[29214]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:59:07.770 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T02:59:08.047 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T02:59:08.459 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:59:08 smithi016 ceph-mon[29214]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:59:08.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:59:08 smithi123 ceph-mon[36358]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:59:09.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:09.591+0000 7ff2e0bb6640 1 -- 172.21.15.16:0/1729270484 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2dc1022c0 msgr2=0x7ff2dc102690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:59:09.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:09.591+0000 7ff2e0bb6640 1 --2- 172.21.15.16:0/1729270484 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2dc1022c0 0x7ff2dc102690 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc009c50 tx=0x7ff2cc004d10 comp rx=0 tx=0).stop 2024-06-18T02:59:09.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:09.592+0000 7ff2e0bb6640 1 -- 172.21.15.16:0/1729270484 shutdown_connections 2024-06-18T02:59:09.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:09.592+0000 7ff2e0bb6640 1 --2- 172.21.15.16:0/1729270484 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2dc102c60 0x7ff2dc0fc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:59:09.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:09.592+0000 7ff2e0bb6640 1 --2- 172.21.15.16:0/1729270484 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2dc1022c0 0x7ff2dc102690 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:59:09.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:09.592+0000 7ff2e0bb6640 1 -- 172.21.15.16:0/1729270484 >> 172.21.15.16:0/1729270484 conn(0x7ff2dc0f7e50 msgr2=0x7ff2dc0fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:59:09.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:09.592+0000 7ff2e0bb6640 1 -- 172.21.15.16:0/1729270484 shutdown_connections 2024-06-18T02:59:09.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:09.592+0000 7ff2e0bb6640 1 -- 172.21.15.16:0/1729270484 wait complete. 2024-06-18T02:59:09.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:09.592+0000 7ff2e0bb6640 1 Processor -- start 2024-06-18T02:59:09.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:09.592+0000 7ff2e0bb6640 1 -- start start 2024-06-18T02:59:09.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:09.592+0000 7ff2e0bb6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2dc1022c0 0x7ff2dc06f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:59:09.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:09.592+0000 7ff2e0bb6640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2dc102c60 0x7ff2dc06fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:59:09.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:09.592+0000 7ff2e0bb6640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2dc0701f0 con 0x7ff2dc1022c0 2024-06-18T02:59:09.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:09.592+0000 7ff2e0bb6640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2dc070360 con 0x7ff2dc102c60 2024-06-18T02:59:09.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:09.593+0000 7ff2da59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2dc1022c0 0x7ff2dc06f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:59:09.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:09.593+0000 7ff2da59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2dc1022c0 0x7ff2dc06f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35496/0 (socket says 172.21.15.16:35496) 2024-06-18T02:59:09.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:09.593+0000 7ff2da59c640 1 -- 172.21.15.16:0/649573756 learned_addr learned my addr 172.21.15.16:0/649573756 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:59:09.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:09.593+0000 7ff2d9d9b640 1 --2- 172.21.15.16:0/649573756 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2dc102c60 0x7ff2dc06fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:59:09.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:09.593+0000 7ff2da59c640 1 -- 172.21.15.16:0/649573756 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2dc102c60 msgr2=0x7ff2dc06fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:59:09.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:09.593+0000 7ff2da59c640 1 --2- 172.21.15.16:0/649573756 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2dc102c60 0x7ff2dc06fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:59:09.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:09.593+0000 7ff2da59c640 1 -- 172.21.15.16:0/649573756 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2cc0098b0 con 0x7ff2dc1022c0 2024-06-18T02:59:09.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:09.593+0000 7ff2da59c640 1 --2- 172.21.15.16:0/649573756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2dc1022c0 0x7ff2dc06f770 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc004750 tx=0x7ff2cc004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:59:09.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:09.593+0000 7ff2d2ffd640 1 -- 172.21.15.16:0/649573756 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff2cc007d50 con 0x7ff2dc1022c0 2024-06-18T02:59:09.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:09.593+0000 7ff2d2ffd640 1 -- 172.21.15.16:0/649573756 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff2cc032c00 con 0x7ff2dc1022c0 2024-06-18T02:59:09.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:09.593+0000 7ff2e0bb6640 1 -- 172.21.15.16:0/649573756 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2dc070590 con 0x7ff2dc1022c0 2024-06-18T02:59:09.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:09.593+0000 7ff2d2ffd640 1 -- 172.21.15.16:0/649573756 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff2cc03da90 con 0x7ff2dc1022c0 2024-06-18T02:59:09.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:09.593+0000 7ff2e0bb6640 1 -- 172.21.15.16:0/649573756 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2dc119930 con 0x7ff2dc1022c0 2024-06-18T02:59:09.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:09.594+0000 7ff2e0bb6640 1 -- 172.21.15.16:0/649573756 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2dc0fda30 con 0x7ff2dc1022c0 2024-06-18T02:59:09.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:09.595+0000 7ff2d2ffd640 1 -- 172.21.15.16:0/649573756 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7ff2cc02f030 con 0x7ff2dc1022c0 2024-06-18T02:59:09.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:09.595+0000 7ff2d2ffd640 1 --2- 172.21.15.16:0/649573756 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7ff2c8065430 0x7ff2c80678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:59:09.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:09.595+0000 7ff2d2ffd640 1 -- 172.21.15.16:0/649573756 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7ff2cc0a34d0 con 0x7ff2dc1022c0 2024-06-18T02:59:09.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:09.595+0000 7ff2d9d9b640 1 --2- 172.21.15.16:0/649573756 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7ff2c8065430 0x7ff2c80678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:59:09.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:09.595+0000 7ff2d9d9b640 1 --2- 172.21.15.16:0/649573756 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7ff2c8065430 0x7ff2c80678e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff2dc0710b0 tx=0x7ff2c400b040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:59:09.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:09.599+0000 7ff2d2ffd640 1 -- 172.21.15.16:0/649573756 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7ff2cc071a00 con 0x7ff2dc1022c0 2024-06-18T02:59:09.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:09.809+0000 7ff2e0bb6640 1 -- 172.21.15.16:0/649573756 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff2dc0fbab0 con 0x7ff2dc1022c0 2024-06-18T02:59:09.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:09.809+0000 7ff2d2ffd640 1 -- 172.21.15.16:0/649573756 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7ff2cc04a100 con 0x7ff2dc1022c0 2024-06-18T02:59:09.810 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:59:09.810 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-18T02:59:09.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:09.811+0000 7ff2e0bb6640 1 -- 172.21.15.16:0/649573756 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7ff2c8065430 msgr2=0x7ff2c80678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:59:09.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:09.811+0000 7ff2e0bb6640 1 --2- 172.21.15.16:0/649573756 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7ff2c8065430 0x7ff2c80678e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff2dc0710b0 tx=0x7ff2c400b040 comp rx=0 tx=0).stop 2024-06-18T02:59:09.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:09.811+0000 7ff2e0bb6640 1 -- 172.21.15.16:0/649573756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2dc1022c0 msgr2=0x7ff2dc06f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:59:09.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:09.811+0000 7ff2e0bb6640 1 --2- 172.21.15.16:0/649573756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2dc1022c0 0x7ff2dc06f770 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc004750 tx=0x7ff2cc004d10 comp rx=0 tx=0).stop 2024-06-18T02:59:09.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:09.811+0000 7ff2e0bb6640 1 -- 172.21.15.16:0/649573756 shutdown_connections 2024-06-18T02:59:09.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:09.811+0000 7ff2e0bb6640 1 --2- 172.21.15.16:0/649573756 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7ff2c8065430 0x7ff2c80678e0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:59:09.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:09.811+0000 7ff2e0bb6640 1 --2- 172.21.15.16:0/649573756 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff2dc102c60 0x7ff2dc06fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:59:09.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:09.811+0000 7ff2e0bb6640 1 --2- 172.21.15.16:0/649573756 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2dc1022c0 0x7ff2dc06f770 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:59:09.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:09.811+0000 7ff2e0bb6640 1 -- 172.21.15.16:0/649573756 >> 172.21.15.16:0/649573756 conn(0x7ff2dc0f7e50 msgr2=0x7ff2dc0f8dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:59:09.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:09.811+0000 7ff2e0bb6640 1 -- 172.21.15.16:0/649573756 shutdown_connections 2024-06-18T02:59:09.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:09.811+0000 7ff2e0bb6640 1 -- 172.21.15.16:0/649573756 wait complete. 2024-06-18T02:59:10.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:59:10 smithi016 ceph-mon[29214]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:59:10.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:59:10 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/649573756' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:59:10.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:59:10 smithi123 ceph-mon[36358]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:59:10.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:59:10 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/649573756' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:59:11.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:59:11 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:59:11.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:59:11 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:59:12.706 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:59:12 smithi016 ceph-mon[29214]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:59:12.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:59:12 smithi123 ceph-mon[36358]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:59:13.484 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T02:59:13.754 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T02:59:14.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:59:14 smithi123 ceph-mon[36358]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:59:14.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:59:14 smithi016 ceph-mon[29214]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:59:15.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:15.300+0000 7f82f4cef640 1 -- 172.21.15.16:0/3408982183 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82f01022c0 msgr2=0x7f82f0102690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:59:15.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:15.300+0000 7f82f4cef640 1 --2- 172.21.15.16:0/3408982183 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82f01022c0 0x7f82f0102690 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f82e0009c50 tx=0x7f82e0004d10 comp rx=0 tx=0).stop 2024-06-18T02:59:15.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:15.300+0000 7f82f4cef640 1 -- 172.21.15.16:0/3408982183 shutdown_connections 2024-06-18T02:59:15.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:15.300+0000 7f82f4cef640 1 --2- 172.21.15.16:0/3408982183 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82f0102c60 0x7f82f00fc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:59:15.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:15.300+0000 7f82f4cef640 1 --2- 172.21.15.16:0/3408982183 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82f01022c0 0x7f82f0102690 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:59:15.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:15.300+0000 7f82f4cef640 1 -- 172.21.15.16:0/3408982183 >> 172.21.15.16:0/3408982183 conn(0x7f82f00f7e50 msgr2=0x7f82f00fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:59:15.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:15.300+0000 7f82f4cef640 1 -- 172.21.15.16:0/3408982183 shutdown_connections 2024-06-18T02:59:15.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:15.300+0000 7f82f4cef640 1 -- 172.21.15.16:0/3408982183 wait complete. 2024-06-18T02:59:15.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:15.300+0000 7f82f4cef640 1 Processor -- start 2024-06-18T02:59:15.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:15.301+0000 7f82f4cef640 1 -- start start 2024-06-18T02:59:15.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:15.301+0000 7f82f4cef640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82f01022c0 0x7f82f006f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:59:15.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:15.301+0000 7f82f4cef640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82f0102c60 0x7f82f006fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:59:15.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:15.301+0000 7f82f4cef640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f00701f0 con 0x7f82f0102c60 2024-06-18T02:59:15.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:15.301+0000 7f82f4cef640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f0070360 con 0x7f82f01022c0 2024-06-18T02:59:15.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:15.301+0000 7f82edd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82f0102c60 0x7f82f006fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:59:15.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:15.301+0000 7f82edd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82f0102c60 0x7f82f006fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:42528/0 (socket says 172.21.15.16:42528) 2024-06-18T02:59:15.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:15.301+0000 7f82edd9b640 1 -- 172.21.15.16:0/1498943899 learned_addr learned my addr 172.21.15.16:0/1498943899 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:59:15.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:15.301+0000 7f82ee59c640 1 --2- 172.21.15.16:0/1498943899 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82f01022c0 0x7f82f006f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:59:15.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:15.301+0000 7f82edd9b640 1 -- 172.21.15.16:0/1498943899 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82f01022c0 msgr2=0x7f82f006f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:59:15.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:15.301+0000 7f82edd9b640 1 --2- 172.21.15.16:0/1498943899 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82f01022c0 0x7f82f006f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:59:15.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:15.301+0000 7f82edd9b640 1 -- 172.21.15.16:0/1498943899 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82e00098b0 con 0x7f82f0102c60 2024-06-18T02:59:15.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:15.301+0000 7f82edd9b640 1 --2- 172.21.15.16:0/1498943899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82f0102c60 0x7f82f006fcb0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f82e400c790 tx=0x7f82e40043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:59:15.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:15.301+0000 7f82deffd640 1 -- 172.21.15.16:0/1498943899 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82e400a820 con 0x7f82f0102c60 2024-06-18T02:59:15.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:15.301+0000 7f82f4cef640 1 -- 172.21.15.16:0/1498943899 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82f00705f0 con 0x7f82f0102c60 2024-06-18T02:59:15.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:15.302+0000 7f82deffd640 1 -- 172.21.15.16:0/1498943899 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f82e400ae50 con 0x7f82f0102c60 2024-06-18T02:59:15.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:15.302+0000 7f82deffd640 1 -- 172.21.15.16:0/1498943899 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82e40115e0 con 0x7f82f0102c60 2024-06-18T02:59:15.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:15.302+0000 7f82f4cef640 1 -- 172.21.15.16:0/1498943899 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82f01199e0 con 0x7f82f0102c60 2024-06-18T02:59:15.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:15.302+0000 7f82f4cef640 1 -- 172.21.15.16:0/1498943899 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82f00fda30 con 0x7f82f0102c60 2024-06-18T02:59:15.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:15.303+0000 7f82deffd640 1 -- 172.21.15.16:0/1498943899 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f82e4012400 con 0x7f82f0102c60 2024-06-18T02:59:15.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:15.303+0000 7f82deffd640 1 --2- 172.21.15.16:0/1498943899 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f82d8065430 0x7f82d80678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:59:15.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:15.303+0000 7f82deffd640 1 -- 172.21.15.16:0/1498943899 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f82e4082690 con 0x7f82f0102c60 2024-06-18T02:59:15.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:15.304+0000 7f82ee59c640 1 --2- 172.21.15.16:0/1498943899 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f82d8065430 0x7f82d80678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:59:15.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:15.305+0000 7f82ee59c640 1 --2- 172.21.15.16:0/1498943899 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f82d8065430 0x7f82d80678e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f82e0003400 tx=0x7f82e0004d10 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:59:15.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:15.308+0000 7f82deffd640 1 -- 172.21.15.16:0/1498943899 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f82e4050bc0 con 0x7f82f0102c60 2024-06-18T02:59:15.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:15.521+0000 7f82f4cef640 1 -- 172.21.15.16:0/1498943899 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f82f01094b0 con 0x7f82f0102c60 2024-06-18T02:59:15.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:15.522+0000 7f82deffd640 1 -- 172.21.15.16:0/1498943899 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f82e40541e0 con 0x7f82f0102c60 2024-06-18T02:59:15.522 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:59:15.522 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-18T02:59:15.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:15.524+0000 7f82f4cef640 1 -- 172.21.15.16:0/1498943899 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f82d8065430 msgr2=0x7f82d80678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:59:15.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:15.524+0000 7f82f4cef640 1 --2- 172.21.15.16:0/1498943899 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f82d8065430 0x7f82d80678e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f82e0003400 tx=0x7f82e0004d10 comp rx=0 tx=0).stop 2024-06-18T02:59:15.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:15.524+0000 7f82f4cef640 1 -- 172.21.15.16:0/1498943899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82f0102c60 msgr2=0x7f82f006fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:59:15.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:15.524+0000 7f82f4cef640 1 --2- 172.21.15.16:0/1498943899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82f0102c60 0x7f82f006fcb0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f82e400c790 tx=0x7f82e40043c0 comp rx=0 tx=0).stop 2024-06-18T02:59:15.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:15.524+0000 7f82f4cef640 1 -- 172.21.15.16:0/1498943899 shutdown_connections 2024-06-18T02:59:15.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:15.524+0000 7f82f4cef640 1 --2- 172.21.15.16:0/1498943899 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f82d8065430 0x7f82d80678e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:59:15.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:15.524+0000 7f82f4cef640 1 --2- 172.21.15.16:0/1498943899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82f0102c60 0x7f82f006fcb0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:59:15.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:15.524+0000 7f82f4cef640 1 --2- 172.21.15.16:0/1498943899 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82f01022c0 0x7f82f006f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:59:15.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:15.524+0000 7f82f4cef640 1 -- 172.21.15.16:0/1498943899 >> 172.21.15.16:0/1498943899 conn(0x7f82f00f7e50 msgr2=0x7f82f00f8dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:59:15.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:15.524+0000 7f82f4cef640 1 -- 172.21.15.16:0/1498943899 shutdown_connections 2024-06-18T02:59:15.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:15.525+0000 7f82f4cef640 1 -- 172.21.15.16:0/1498943899 wait complete. 2024-06-18T02:59:16.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:59:16 smithi016 ceph-mon[29214]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:59:16.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:59:16 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/1498943899' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:59:16.700 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:59:16 smithi123 ceph-mon[36358]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:59:16.700 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:59:16 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/1498943899' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:59:18.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:59:18 smithi123 ceph-mon[36358]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:59:18.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:59:18 smithi016 ceph-mon[29214]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:59:19.182 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T02:59:19.461 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T02:59:20.732 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:59:20 smithi016 ceph-mon[29214]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:59:20.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:59:20 smithi123 ceph-mon[36358]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:59:21.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:21.067+0000 7f776c4f4640 1 -- 172.21.15.16:0/455549437 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7764101ba0 msgr2=0x7f77640fb230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:59:21.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:21.067+0000 7f776c4f4640 1 --2- 172.21.15.16:0/455549437 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7764101ba0 0x7f77640fb230 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f77540048d0 tx=0x7f7754009da0 comp rx=0 tx=0).stop 2024-06-18T02:59:21.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:21.067+0000 7f776c4f4640 1 -- 172.21.15.16:0/455549437 shutdown_connections 2024-06-18T02:59:21.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:21.067+0000 7f776c4f4640 1 --2- 172.21.15.16:0/455549437 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7764101ba0 0x7f77640fb230 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:59:21.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:21.067+0000 7f776c4f4640 1 --2- 172.21.15.16:0/455549437 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7764101200 0x7f77641015d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:59:21.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:21.067+0000 7f776c4f4640 1 -- 172.21.15.16:0/455549437 >> 172.21.15.16:0/455549437 conn(0x7f77640f6df0 msgr2=0x7f77640f9200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:59:21.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:21.067+0000 7f776c4f4640 1 -- 172.21.15.16:0/455549437 shutdown_connections 2024-06-18T02:59:21.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:21.067+0000 7f776c4f4640 1 -- 172.21.15.16:0/455549437 wait complete. 2024-06-18T02:59:21.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:21.067+0000 7f776c4f4640 1 Processor -- start 2024-06-18T02:59:21.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:21.067+0000 7f776c4f4640 1 -- start start 2024-06-18T02:59:21.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:21.068+0000 7f776c4f4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7764101200 0x7f7764111f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:59:21.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:21.068+0000 7f776c4f4640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7764101ba0 0x7f7764112490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:59:21.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:21.068+0000 7f776c4f4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7764112a90 con 0x7f7764101200 2024-06-18T02:59:21.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:21.068+0000 7f776c4f4640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f776410bed0 con 0x7f7764101ba0 2024-06-18T02:59:21.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:21.068+0000 7f776a290640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7764101200 0x7f7764111f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:59:21.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:21.068+0000 7f776a290640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7764101200 0x7f7764111f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36096/0 (socket says 172.21.15.16:36096) 2024-06-18T02:59:21.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:21.068+0000 7f776a290640 1 -- 172.21.15.16:0/3407023744 learned_addr learned my addr 172.21.15.16:0/3407023744 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:59:21.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:21.068+0000 7f7769a8f640 1 --2- 172.21.15.16:0/3407023744 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7764101ba0 0x7f7764112490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:59:21.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:21.068+0000 7f776a290640 1 -- 172.21.15.16:0/3407023744 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7764101ba0 msgr2=0x7f7764112490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:59:21.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:21.068+0000 7f776a290640 1 --2- 172.21.15.16:0/3407023744 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7764101ba0 0x7f7764112490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:59:21.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:21.068+0000 7f776a290640 1 -- 172.21.15.16:0/3407023744 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7754005040 con 0x7f7764101200 2024-06-18T02:59:21.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:21.068+0000 7f776a290640 1 --2- 172.21.15.16:0/3407023744 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7764101200 0x7f7764111f50 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f7760002a50 tx=0x7f7760004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:59:21.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:21.068+0000 7f775affd640 1 -- 172.21.15.16:0/3407023744 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7760007c30 con 0x7f7764101200 2024-06-18T02:59:21.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:21.068+0000 7f775affd640 1 -- 172.21.15.16:0/3407023744 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f776000f040 con 0x7f7764101200 2024-06-18T02:59:21.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:21.068+0000 7f776c4f4640 1 -- 172.21.15.16:0/3407023744 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f776410c1b0 con 0x7f7764101200 2024-06-18T02:59:21.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:21.068+0000 7f775affd640 1 -- 172.21.15.16:0/3407023744 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f776000dad0 con 0x7f7764101200 2024-06-18T02:59:21.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:21.069+0000 7f776c4f4640 1 -- 172.21.15.16:0/3407023744 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f776410c720 con 0x7f7764101200 2024-06-18T02:59:21.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:21.070+0000 7f776c4f4640 1 -- 172.21.15.16:0/3407023744 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77640fc970 con 0x7f7764101200 2024-06-18T02:59:21.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:21.070+0000 7f775affd640 1 -- 172.21.15.16:0/3407023744 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f776000dc70 con 0x7f7764101200 2024-06-18T02:59:21.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:21.070+0000 7f775affd640 1 --2- 172.21.15.16:0/3407023744 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f7750065430 0x7f77500678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:59:21.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:21.071+0000 7f775affd640 1 -- 172.21.15.16:0/3407023744 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f7760081bb0 con 0x7f7764101200 2024-06-18T02:59:21.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:21.071+0000 7f7769a8f640 1 --2- 172.21.15.16:0/3407023744 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f7750065430 0x7f77500678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:59:21.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:21.071+0000 7f7769a8f640 1 --2- 172.21.15.16:0/3407023744 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f7750065430 0x7f77500678e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7754009f80 tx=0x7f7754035040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:59:21.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:21.075+0000 7f775affd640 1 -- 172.21.15.16:0/3407023744 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f776004cdf0 con 0x7f7764101200 2024-06-18T02:59:21.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:21.284+0000 7f776c4f4640 1 -- 172.21.15.16:0/3407023744 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f7764108360 con 0x7f7764101200 2024-06-18T02:59:21.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:21.284+0000 7f775affd640 1 -- 172.21.15.16:0/3407023744 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f7760088080 con 0x7f7764101200 2024-06-18T02:59:21.285 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:59:21.285 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-18T02:59:21.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:21.287+0000 7f776c4f4640 1 -- 172.21.15.16:0/3407023744 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f7750065430 msgr2=0x7f77500678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:59:21.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:21.287+0000 7f776c4f4640 1 --2- 172.21.15.16:0/3407023744 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f7750065430 0x7f77500678e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7754009f80 tx=0x7f7754035040 comp rx=0 tx=0).stop 2024-06-18T02:59:21.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:21.287+0000 7f776c4f4640 1 -- 172.21.15.16:0/3407023744 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7764101200 msgr2=0x7f7764111f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:59:21.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:21.287+0000 7f776c4f4640 1 --2- 172.21.15.16:0/3407023744 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7764101200 0x7f7764111f50 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f7760002a50 tx=0x7f7760004600 comp rx=0 tx=0).stop 2024-06-18T02:59:21.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:21.287+0000 7f776c4f4640 1 -- 172.21.15.16:0/3407023744 shutdown_connections 2024-06-18T02:59:21.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:21.287+0000 7f776c4f4640 1 --2- 172.21.15.16:0/3407023744 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f7750065430 0x7f77500678e0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:59:21.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:21.287+0000 7f776c4f4640 1 --2- 172.21.15.16:0/3407023744 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7764101ba0 0x7f7764112490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:59:21.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:21.287+0000 7f776c4f4640 1 --2- 172.21.15.16:0/3407023744 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7764101200 0x7f7764111f50 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:59:21.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:21.287+0000 7f776c4f4640 1 -- 172.21.15.16:0/3407023744 >> 172.21.15.16:0/3407023744 conn(0x7f77640f6df0 msgr2=0x7f77640f7d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:59:21.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:21.287+0000 7f776c4f4640 1 -- 172.21.15.16:0/3407023744 shutdown_connections 2024-06-18T02:59:21.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:21.287+0000 7f776c4f4640 1 -- 172.21.15.16:0/3407023744 wait complete. 2024-06-18T02:59:21.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:59:21 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/3407023744' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:59:21.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:59:21 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/3407023744' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:59:22.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:59:22 smithi123 ceph-mon[36358]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:59:22.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:59:22 smithi016 ceph-mon[29214]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:59:24.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:59:24 smithi016 ceph-mon[29214]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:59:24.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:59:24 smithi123 ceph-mon[36358]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:59:24.947 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T02:59:25.225 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T02:59:26.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:59:26 smithi016 ceph-mon[29214]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:59:26.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:59:26 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:59:26.702 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:59:26 smithi123 ceph-mon[36358]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:59:26.702 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:59:26 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:59:26.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:26.826+0000 7f26634d5640 1 -- 172.21.15.16:0/3630326116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f265c102c40 msgr2=0x7f265c0fc2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:59:26.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:26.826+0000 7f26634d5640 1 --2- 172.21.15.16:0/3630326116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f265c102c40 0x7f265c0fc2d0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f264c0048d0 tx=0x7f264c009da0 comp rx=0 tx=0).stop 2024-06-18T02:59:26.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:26.826+0000 7f26634d5640 1 -- 172.21.15.16:0/3630326116 shutdown_connections 2024-06-18T02:59:26.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:26.826+0000 7f26634d5640 1 --2- 172.21.15.16:0/3630326116 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f265c102c40 0x7f265c0fc2d0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:59:26.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:26.826+0000 7f26634d5640 1 --2- 172.21.15.16:0/3630326116 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f265c1022a0 0x7f265c102670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:59:26.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:26.826+0000 7f26634d5640 1 -- 172.21.15.16:0/3630326116 >> 172.21.15.16:0/3630326116 conn(0x7f265c0f7e50 msgr2=0x7f265c0fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:59:26.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:26.826+0000 7f26634d5640 1 -- 172.21.15.16:0/3630326116 shutdown_connections 2024-06-18T02:59:26.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:26.826+0000 7f26634d5640 1 -- 172.21.15.16:0/3630326116 wait complete. 2024-06-18T02:59:26.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:26.827+0000 7f26634d5640 1 Processor -- start 2024-06-18T02:59:26.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:26.827+0000 7f26634d5640 1 -- start start 2024-06-18T02:59:26.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:26.827+0000 7f26634d5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f265c1022a0 0x7f265c113020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:59:26.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:26.827+0000 7f26634d5640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f265c102c40 0x7f265c113560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:59:26.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:26.827+0000 7f26634d5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f265c113b60 con 0x7f265c1022a0 2024-06-18T02:59:26.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:26.827+0000 7f26634d5640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f265c10cfa0 con 0x7f265c102c40 2024-06-18T02:59:26.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:26.827+0000 7f2661271640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f265c1022a0 0x7f265c113020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:59:26.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:26.827+0000 7f2661271640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f265c1022a0 0x7f265c113020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36116/0 (socket says 172.21.15.16:36116) 2024-06-18T02:59:26.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:26.827+0000 7f2661271640 1 -- 172.21.15.16:0/2422938090 learned_addr learned my addr 172.21.15.16:0/2422938090 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:59:26.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:26.827+0000 7f2660a70640 1 --2- 172.21.15.16:0/2422938090 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f265c102c40 0x7f265c113560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:59:26.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:26.827+0000 7f2661271640 1 -- 172.21.15.16:0/2422938090 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f265c102c40 msgr2=0x7f265c113560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:59:26.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:26.827+0000 7f2661271640 1 --2- 172.21.15.16:0/2422938090 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f265c102c40 0x7f265c113560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:59:26.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:26.827+0000 7f2661271640 1 -- 172.21.15.16:0/2422938090 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f264c005040 con 0x7f265c1022a0 2024-06-18T02:59:26.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:26.827+0000 7f2660a70640 1 --2- 172.21.15.16:0/2422938090 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f265c102c40 0x7f265c113560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-18T02:59:26.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:26.828+0000 7f2661271640 1 --2- 172.21.15.16:0/2422938090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f265c1022a0 0x7f265c113020 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f265800eb70 tx=0x7f265800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:59:26.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:26.828+0000 7f2651ffb640 1 -- 172.21.15.16:0/2422938090 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2658007c30 con 0x7f265c1022a0 2024-06-18T02:59:26.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:26.828+0000 7f2651ffb640 1 -- 172.21.15.16:0/2422938090 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f265800f040 con 0x7f265c1022a0 2024-06-18T02:59:26.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:26.828+0000 7f26634d5640 1 -- 172.21.15.16:0/2422938090 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f265c10d220 con 0x7f265c1022a0 2024-06-18T02:59:26.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:26.828+0000 7f2651ffb640 1 -- 172.21.15.16:0/2422938090 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f26580095b0 con 0x7f265c1022a0 2024-06-18T02:59:26.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:26.828+0000 7f26634d5640 1 -- 172.21.15.16:0/2422938090 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f265c10d7c0 con 0x7f265c1022a0 2024-06-18T02:59:26.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:26.829+0000 7f26634d5640 1 -- 172.21.15.16:0/2422938090 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f265c0fda10 con 0x7f265c1022a0 2024-06-18T02:59:26.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:26.829+0000 7f2651ffb640 1 -- 172.21.15.16:0/2422938090 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f2658009730 con 0x7f265c1022a0 2024-06-18T02:59:26.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:26.830+0000 7f2651ffb640 1 --2- 172.21.15.16:0/2422938090 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f2648065430 0x7f26480678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:59:26.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:26.830+0000 7f2651ffb640 1 -- 172.21.15.16:0/2422938090 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f26580821f0 con 0x7f265c1022a0 2024-06-18T02:59:26.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:26.830+0000 7f2660a70640 1 --2- 172.21.15.16:0/2422938090 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f2648065430 0x7f26480678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:59:26.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:26.830+0000 7f2660a70640 1 --2- 172.21.15.16:0/2422938090 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f2648065430 0x7f26480678e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f264c009f80 tx=0x7f264c009da0 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:59:26.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:26.834+0000 7f2651ffb640 1 -- 172.21.15.16:0/2422938090 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f26580506a0 con 0x7f265c1022a0 2024-06-18T02:59:27.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:27.042+0000 7f26634d5640 1 -- 172.21.15.16:0/2422938090 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f265c109400 con 0x7f265c1022a0 2024-06-18T02:59:27.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:27.043+0000 7f2651ffb640 1 -- 172.21.15.16:0/2422938090 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f2658053cc0 con 0x7f265c1022a0 2024-06-18T02:59:27.043 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:59:27.043 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-18T02:59:27.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:27.044+0000 7f26634d5640 1 -- 172.21.15.16:0/2422938090 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f2648065430 msgr2=0x7f26480678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:59:27.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:27.044+0000 7f26634d5640 1 --2- 172.21.15.16:0/2422938090 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f2648065430 0x7f26480678e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f264c009f80 tx=0x7f264c009da0 comp rx=0 tx=0).stop 2024-06-18T02:59:27.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:27.044+0000 7f26634d5640 1 -- 172.21.15.16:0/2422938090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f265c1022a0 msgr2=0x7f265c113020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:59:27.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:27.044+0000 7f26634d5640 1 --2- 172.21.15.16:0/2422938090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f265c1022a0 0x7f265c113020 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f265800eb70 tx=0x7f265800b040 comp rx=0 tx=0).stop 2024-06-18T02:59:27.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:27.045+0000 7f26634d5640 1 -- 172.21.15.16:0/2422938090 shutdown_connections 2024-06-18T02:59:27.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:27.045+0000 7f26634d5640 1 --2- 172.21.15.16:0/2422938090 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f2648065430 0x7f26480678e0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:59:27.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:27.045+0000 7f26634d5640 1 --2- 172.21.15.16:0/2422938090 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f265c102c40 0x7f265c113560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:59:27.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:27.045+0000 7f26634d5640 1 --2- 172.21.15.16:0/2422938090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f265c1022a0 0x7f265c113020 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:59:27.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:27.045+0000 7f26634d5640 1 -- 172.21.15.16:0/2422938090 >> 172.21.15.16:0/2422938090 conn(0x7f265c0f7e50 msgr2=0x7f265c0f8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:59:27.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:27.045+0000 7f26634d5640 1 -- 172.21.15.16:0/2422938090 shutdown_connections 2024-06-18T02:59:27.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:27.045+0000 7f26634d5640 1 -- 172.21.15.16:0/2422938090 wait complete. 2024-06-18T02:59:27.636 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:59:27 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/2422938090' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:59:27.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:59:27 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/2422938090' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:59:28.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:59:28 smithi123 ceph-mon[36358]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:59:28.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:59:28 smithi016 ceph-mon[29214]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:59:30.662 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T02:59:30.691 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:59:30 smithi016 ceph-mon[29214]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:59:30.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:59:30 smithi123 ceph-mon[36358]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:59:30.939 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T02:59:32.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:32.566+0000 7f6675e9d640 1 -- 172.21.15.16:0/3097566480 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6670101b10 msgr2=0x7f66700fb1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:59:32.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:32.566+0000 7f6675e9d640 1 --2- 172.21.15.16:0/3097566480 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6670101b10 0x7f66700fb1a0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f66580048d0 tx=0x7f6658009da0 comp rx=0 tx=0).stop 2024-06-18T02:59:32.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:32.566+0000 7f6675e9d640 1 -- 172.21.15.16:0/3097566480 shutdown_connections 2024-06-18T02:59:32.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:32.566+0000 7f6675e9d640 1 --2- 172.21.15.16:0/3097566480 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6670101b10 0x7f66700fb1a0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:59:32.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:32.566+0000 7f6675e9d640 1 --2- 172.21.15.16:0/3097566480 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6670101170 0x7f6670101540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:59:32.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:32.566+0000 7f6675e9d640 1 -- 172.21.15.16:0/3097566480 >> 172.21.15.16:0/3097566480 conn(0x7f66700f6d80 msgr2=0x7f66700f9190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:59:32.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:32.566+0000 7f6675e9d640 1 -- 172.21.15.16:0/3097566480 shutdown_connections 2024-06-18T02:59:32.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:32.566+0000 7f6675e9d640 1 -- 172.21.15.16:0/3097566480 wait complete. 2024-06-18T02:59:32.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:32.566+0000 7f6675e9d640 1 Processor -- start 2024-06-18T02:59:32.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:32.566+0000 7f6675e9d640 1 -- start start 2024-06-18T02:59:32.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:32.567+0000 7f6675e9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6670101170 0x7f6670112f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:59:32.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:32.567+0000 7f6675e9d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6670101b10 0x7f66701134a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:59:32.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:32.567+0000 7f6675e9d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6670113aa0 con 0x7f6670101170 2024-06-18T02:59:32.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:32.567+0000 7f6675e9d640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6670113c10 con 0x7f6670101b10 2024-06-18T02:59:32.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:32.567+0000 7f666f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6670101170 0x7f6670112f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:59:32.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:32.567+0000 7f666f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6670101170 0x7f6670112f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39232/0 (socket says 172.21.15.16:39232) 2024-06-18T02:59:32.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:32.567+0000 7f666f7fe640 1 -- 172.21.15.16:0/396431605 learned_addr learned my addr 172.21.15.16:0/396431605 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:59:32.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:32.567+0000 7f6666ffd640 1 --2- 172.21.15.16:0/396431605 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6670101b10 0x7f66701134a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:59:32.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:32.567+0000 7f666f7fe640 1 -- 172.21.15.16:0/396431605 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6670101b10 msgr2=0x7f66701134a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:59:32.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:32.567+0000 7f666f7fe640 1 --2- 172.21.15.16:0/396431605 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6670101b10 0x7f66701134a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:59:32.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:32.567+0000 7f666f7fe640 1 -- 172.21.15.16:0/396431605 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6658005040 con 0x7f6670101170 2024-06-18T02:59:32.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:32.567+0000 7f666f7fe640 1 --2- 172.21.15.16:0/396431605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6670101170 0x7f6670112f60 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f6660002a50 tx=0x7f6660004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:59:32.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:32.567+0000 7f666cff9640 1 -- 172.21.15.16:0/396431605 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6660007c30 con 0x7f6670101170 2024-06-18T02:59:32.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:32.567+0000 7f666cff9640 1 -- 172.21.15.16:0/396431605 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f666000f040 con 0x7f6670101170 2024-06-18T02:59:32.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:32.567+0000 7f6675e9d640 1 -- 172.21.15.16:0/396431605 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f667010d090 con 0x7f6670101170 2024-06-18T02:59:32.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:32.567+0000 7f666cff9640 1 -- 172.21.15.16:0/396431605 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f666000db60 con 0x7f6670101170 2024-06-18T02:59:32.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:32.567+0000 7f6675e9d640 1 -- 172.21.15.16:0/396431605 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f667010d600 con 0x7f6670101170 2024-06-18T02:59:32.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:32.569+0000 7f6675e9d640 1 -- 172.21.15.16:0/396431605 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66700fc8e0 con 0x7f6670101170 2024-06-18T02:59:32.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:32.569+0000 7f666cff9640 1 -- 172.21.15.16:0/396431605 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f666000dcc0 con 0x7f6670101170 2024-06-18T02:59:32.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:32.569+0000 7f666cff9640 1 --2- 172.21.15.16:0/396431605 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f665c065440 0x7f665c0678f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:59:32.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:32.569+0000 7f666cff9640 1 -- 172.21.15.16:0/396431605 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f6660082240 con 0x7f6670101170 2024-06-18T02:59:32.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:32.570+0000 7f6666ffd640 1 --2- 172.21.15.16:0/396431605 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f665c065440 0x7f665c0678f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:59:32.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:32.570+0000 7f6666ffd640 1 --2- 172.21.15.16:0/396431605 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f665c065440 0x7f665c0678f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6658032430 tx=0x7f6658035040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:59:32.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:32.574+0000 7f666cff9640 1 -- 172.21.15.16:0/396431605 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f6660050830 con 0x7f6670101170 2024-06-18T02:59:32.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:59:32 smithi123 ceph-mon[36358]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:59:32.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:32.782+0000 7f6675e9d640 1 -- 172.21.15.16:0/396431605 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f667010e610 con 0x7f6670101170 2024-06-18T02:59:32.783 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:59:32 smithi016 ceph-mon[29214]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:59:32.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:32.783+0000 7f666cff9640 1 -- 172.21.15.16:0/396431605 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f6660053e50 con 0x7f6670101170 2024-06-18T02:59:32.783 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:59:32.783 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-18T02:59:32.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:32.785+0000 7f6675e9d640 1 -- 172.21.15.16:0/396431605 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f665c065440 msgr2=0x7f665c0678f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:59:32.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:32.785+0000 7f6675e9d640 1 --2- 172.21.15.16:0/396431605 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f665c065440 0x7f665c0678f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6658032430 tx=0x7f6658035040 comp rx=0 tx=0).stop 2024-06-18T02:59:32.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:32.785+0000 7f6675e9d640 1 -- 172.21.15.16:0/396431605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6670101170 msgr2=0x7f6670112f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:59:32.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:32.785+0000 7f6675e9d640 1 --2- 172.21.15.16:0/396431605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6670101170 0x7f6670112f60 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f6660002a50 tx=0x7f6660004600 comp rx=0 tx=0).stop 2024-06-18T02:59:32.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:32.785+0000 7f6675e9d640 1 -- 172.21.15.16:0/396431605 shutdown_connections 2024-06-18T02:59:32.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:32.785+0000 7f6675e9d640 1 --2- 172.21.15.16:0/396431605 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f665c065440 0x7f665c0678f0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:59:32.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:32.785+0000 7f6675e9d640 1 --2- 172.21.15.16:0/396431605 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6670101b10 0x7f66701134a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:59:32.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:32.785+0000 7f6675e9d640 1 --2- 172.21.15.16:0/396431605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6670101170 0x7f6670112f60 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:59:32.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:32.785+0000 7f6675e9d640 1 -- 172.21.15.16:0/396431605 >> 172.21.15.16:0/396431605 conn(0x7f66700f6d80 msgr2=0x7f6670065570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:59:32.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:32.785+0000 7f6675e9d640 1 -- 172.21.15.16:0/396431605 shutdown_connections 2024-06-18T02:59:32.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:32.785+0000 7f6675e9d640 1 -- 172.21.15.16:0/396431605 wait complete. 2024-06-18T02:59:33.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:59:33 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/396431605' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:59:33.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:59:33 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/396431605' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:59:34.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:59:34 smithi016 ceph-mon[29214]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:59:34.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:59:34 smithi123 ceph-mon[36358]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:59:36.607 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T02:59:36.699 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:59:36 smithi123 ceph-mon[36358]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:59:36.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:59:36 smithi016 ceph-mon[29214]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:59:36.888 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T02:59:37.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:59:37 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:59:37.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:59:37 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:59:37.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:59:37 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T02:59:37.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:59:37 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:59:37.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:59:37 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T02:59:37.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:59:37 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T02:59:37.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:59:37 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T02:59:37.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:59:37 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T02:59:38.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:38.478+0000 7f5b0e2fc640 1 -- 172.21.15.16:0/3996325833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b080fa9c0 msgr2=0x7f5b080fae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:59:38.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:38.478+0000 7f5b0e2fc640 1 --2- 172.21.15.16:0/3996325833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b080fa9c0 0x7f5b080fae30 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f5afc0048d0 tx=0x7f5afc009da0 comp rx=0 tx=0).stop 2024-06-18T02:59:38.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:38.478+0000 7f5b0e2fc640 1 -- 172.21.15.16:0/3996325833 shutdown_connections 2024-06-18T02:59:38.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:38.478+0000 7f5b0e2fc640 1 --2- 172.21.15.16:0/3996325833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b080fa9c0 0x7f5b080fae30 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:59:38.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:38.478+0000 7f5b0e2fc640 1 --2- 172.21.15.16:0/3996325833 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b08102370 0x7f5b08102740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:59:38.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:38.478+0000 7f5b0e2fc640 1 -- 172.21.15.16:0/3996325833 >> 172.21.15.16:0/3996325833 conn(0x7f5b080709e0 msgr2=0x7f5b08070de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:59:38.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:38.478+0000 7f5b0e2fc640 1 -- 172.21.15.16:0/3996325833 shutdown_connections 2024-06-18T02:59:38.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:38.478+0000 7f5b0e2fc640 1 -- 172.21.15.16:0/3996325833 wait complete. 2024-06-18T02:59:38.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:38.479+0000 7f5b0e2fc640 1 Processor -- start 2024-06-18T02:59:38.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:38.479+0000 7f5b0e2fc640 1 -- start start 2024-06-18T02:59:38.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:38.479+0000 7f5b0e2fc640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b080fa9c0 0x7f5b08112fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:59:38.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:38.479+0000 7f5b0e2fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b08102370 0x7f5b08113500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:59:38.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:38.479+0000 7f5b0e2fc640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b08113b00 con 0x7f5b08102370 2024-06-18T02:59:38.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:38.479+0000 7f5b0e2fc640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b0810cf40 con 0x7f5b080fa9c0 2024-06-18T02:59:38.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:38.479+0000 7f5b077fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b08102370 0x7f5b08113500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:59:38.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:38.479+0000 7f5b077fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b08102370 0x7f5b08113500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39256/0 (socket says 172.21.15.16:39256) 2024-06-18T02:59:38.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:38.479+0000 7f5b077fe640 1 -- 172.21.15.16:0/2839016609 learned_addr learned my addr 172.21.15.16:0/2839016609 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:59:38.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:38.480+0000 7f5b07fff640 1 --2- 172.21.15.16:0/2839016609 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b080fa9c0 0x7f5b08112fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:59:38.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:38.480+0000 7f5b077fe640 1 -- 172.21.15.16:0/2839016609 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b080fa9c0 msgr2=0x7f5b08112fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:59:38.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:38.480+0000 7f5b077fe640 1 --2- 172.21.15.16:0/2839016609 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b080fa9c0 0x7f5b08112fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:59:38.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:38.480+0000 7f5b077fe640 1 -- 172.21.15.16:0/2839016609 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5afc005040 con 0x7f5b08102370 2024-06-18T02:59:38.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:38.480+0000 7f5b07fff640 1 --2- 172.21.15.16:0/2839016609 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b080fa9c0 0x7f5b08112fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-18T02:59:38.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:38.480+0000 7f5b077fe640 1 --2- 172.21.15.16:0/2839016609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b08102370 0x7f5b08113500 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f5afc003080 tx=0x7f5afc009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:59:38.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:38.480+0000 7f5b04ff9640 1 -- 172.21.15.16:0/2839016609 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5afc038070 con 0x7f5b08102370 2024-06-18T02:59:38.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:38.480+0000 7f5b04ff9640 1 -- 172.21.15.16:0/2839016609 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5afc032970 con 0x7f5b08102370 2024-06-18T02:59:38.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:38.480+0000 7f5b0e2fc640 1 -- 172.21.15.16:0/2839016609 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b0810d1c0 con 0x7f5b08102370 2024-06-18T02:59:38.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:38.480+0000 7f5b04ff9640 1 -- 172.21.15.16:0/2839016609 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5afc03d960 con 0x7f5b08102370 2024-06-18T02:59:38.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:38.480+0000 7f5b0e2fc640 1 -- 172.21.15.16:0/2839016609 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b0810d700 con 0x7f5b08102370 2024-06-18T02:59:38.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:38.481+0000 7f5b0e2fc640 1 -- 172.21.15.16:0/2839016609 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b08065570 con 0x7f5b08102370 2024-06-18T02:59:38.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:38.482+0000 7f5b04ff9640 1 -- 172.21.15.16:0/2839016609 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f5afc030020 con 0x7f5b08102370 2024-06-18T02:59:38.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:38.482+0000 7f5b04ff9640 1 --2- 172.21.15.16:0/2839016609 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f5af0065430 0x7f5af00678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:59:38.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:38.482+0000 7f5b04ff9640 1 -- 172.21.15.16:0/2839016609 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f5afc0a2630 con 0x7f5b08102370 2024-06-18T02:59:38.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:38.482+0000 7f5b07fff640 1 --2- 172.21.15.16:0/2839016609 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f5af0065430 0x7f5af00678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:59:38.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:38.483+0000 7f5b07fff640 1 --2- 172.21.15.16:0/2839016609 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f5af0065430 0x7f5af00678e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5af8004ab0 tx=0x7f5af800b040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:59:38.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:38.487+0000 7f5b04ff9640 1 -- 172.21.15.16:0/2839016609 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f5afc070ae0 con 0x7f5b08102370 2024-06-18T02:59:38.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:59:38 smithi016 ceph-mon[29214]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:59:38.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:38.699+0000 7f5b0e2fc640 1 -- 172.21.15.16:0/2839016609 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f5b081094d0 con 0x7f5b08102370 2024-06-18T02:59:38.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:38.699+0000 7f5b04ff9640 1 -- 172.21.15.16:0/2839016609 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f5afc074100 con 0x7f5b08102370 2024-06-18T02:59:38.700 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:59:38.700 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-18T02:59:38.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:38.701+0000 7f5b0e2fc640 1 -- 172.21.15.16:0/2839016609 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f5af0065430 msgr2=0x7f5af00678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:59:38.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:38.701+0000 7f5b0e2fc640 1 --2- 172.21.15.16:0/2839016609 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f5af0065430 0x7f5af00678e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5af8004ab0 tx=0x7f5af800b040 comp rx=0 tx=0).stop 2024-06-18T02:59:38.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:38.701+0000 7f5b0e2fc640 1 -- 172.21.15.16:0/2839016609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b08102370 msgr2=0x7f5b08113500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:59:38.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:38.701+0000 7f5b0e2fc640 1 --2- 172.21.15.16:0/2839016609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b08102370 0x7f5b08113500 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f5afc003080 tx=0x7f5afc009da0 comp rx=0 tx=0).stop 2024-06-18T02:59:38.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:38.701+0000 7f5b0e2fc640 1 -- 172.21.15.16:0/2839016609 shutdown_connections 2024-06-18T02:59:38.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:38.701+0000 7f5b0e2fc640 1 --2- 172.21.15.16:0/2839016609 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f5af0065430 0x7f5af00678e0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:59:38.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:38.701+0000 7f5b0e2fc640 1 --2- 172.21.15.16:0/2839016609 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b08102370 0x7f5b08113500 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:59:38.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:38.701+0000 7f5b0e2fc640 1 --2- 172.21.15.16:0/2839016609 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b080fa9c0 0x7f5b08112fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:59:38.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:38.701+0000 7f5b0e2fc640 1 -- 172.21.15.16:0/2839016609 >> 172.21.15.16:0/2839016609 conn(0x7f5b080709e0 msgr2=0x7f5b080f9c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:59:38.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:38.701+0000 7f5b0e2fc640 1 -- 172.21.15.16:0/2839016609 shutdown_connections 2024-06-18T02:59:38.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:38.701+0000 7f5b0e2fc640 1 -- 172.21.15.16:0/2839016609 wait complete. 2024-06-18T02:59:38.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:59:38 smithi123 ceph-mon[36358]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:59:39.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:59:39 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/2839016609' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:59:39.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:59:39 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/2839016609' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:59:40.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:59:40 smithi123 ceph-mon[36358]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:59:40.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:59:40 smithi016 ceph-mon[29214]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:59:41.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:59:41 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:59:41.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:59:41 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:59:42.335 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T02:59:42.612 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T02:59:42.706 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:59:42 smithi016 ceph-mon[29214]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:59:42.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:59:42 smithi123 ceph-mon[36358]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:59:44.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:44.283+0000 7fd575cb1640 1 -- 172.21.15.16:0/1522522472 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd570065010 msgr2=0x7fd5700653e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:59:44.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:44.283+0000 7fd575cb1640 1 --2- 172.21.15.16:0/1522522472 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd570065010 0x7fd5700653e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fd5600048d0 tx=0x7fd560009da0 comp rx=0 tx=0).stop 2024-06-18T02:59:44.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:44.283+0000 7fd575cb1640 1 -- 172.21.15.16:0/1522522472 shutdown_connections 2024-06-18T02:59:44.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:44.283+0000 7fd575cb1640 1 --2- 172.21.15.16:0/1522522472 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd5700fdeb0 0x7fd5700fe320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:59:44.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:44.283+0000 7fd575cb1640 1 --2- 172.21.15.16:0/1522522472 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd570065010 0x7fd5700653e0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:59:44.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:44.283+0000 7fd575cb1640 1 -- 172.21.15.16:0/1522522472 >> 172.21.15.16:0/1522522472 conn(0x7fd570072110 msgr2=0x7fd570072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:59:44.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:44.283+0000 7fd575cb1640 1 -- 172.21.15.16:0/1522522472 shutdown_connections 2024-06-18T02:59:44.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:44.283+0000 7fd575cb1640 1 -- 172.21.15.16:0/1522522472 wait complete. 2024-06-18T02:59:44.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:44.284+0000 7fd575cb1640 1 Processor -- start 2024-06-18T02:59:44.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:44.284+0000 7fd575cb1640 1 -- start start 2024-06-18T02:59:44.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:44.284+0000 7fd575cb1640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd570065010 0x7fd570113020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:59:44.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:44.284+0000 7fd575cb1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5700fdeb0 0x7fd570113560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:59:44.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:44.284+0000 7fd575cb1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd570113b60 con 0x7fd5700fdeb0 2024-06-18T02:59:44.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:44.284+0000 7fd575cb1640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd57010cfa0 con 0x7fd570065010 2024-06-18T02:59:44.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:44.284+0000 7fd56effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5700fdeb0 0x7fd570113560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:59:44.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:44.284+0000 7fd56effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5700fdeb0 0x7fd570113560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33500/0 (socket says 172.21.15.16:33500) 2024-06-18T02:59:44.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:44.284+0000 7fd56effd640 1 -- 172.21.15.16:0/3062878508 learned_addr learned my addr 172.21.15.16:0/3062878508 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:59:44.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:44.284+0000 7fd56f7fe640 1 --2- 172.21.15.16:0/3062878508 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd570065010 0x7fd570113020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:59:44.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:44.284+0000 7fd56effd640 1 -- 172.21.15.16:0/3062878508 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd570065010 msgr2=0x7fd570113020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:59:44.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:44.284+0000 7fd56effd640 1 --2- 172.21.15.16:0/3062878508 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd570065010 0x7fd570113020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:59:44.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:44.284+0000 7fd56effd640 1 -- 172.21.15.16:0/3062878508 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd560005040 con 0x7fd5700fdeb0 2024-06-18T02:59:44.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:44.285+0000 7fd56effd640 1 --2- 172.21.15.16:0/3062878508 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5700fdeb0 0x7fd570113560 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fd56400eb70 tx=0x7fd56400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:59:44.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:44.285+0000 7fd557fff640 1 -- 172.21.15.16:0/3062878508 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd564007c30 con 0x7fd5700fdeb0 2024-06-18T02:59:44.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:44.285+0000 7fd557fff640 1 -- 172.21.15.16:0/3062878508 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd56400f040 con 0x7fd5700fdeb0 2024-06-18T02:59:44.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:44.285+0000 7fd575cb1640 1 -- 172.21.15.16:0/3062878508 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd57010d220 con 0x7fd5700fdeb0 2024-06-18T02:59:44.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:44.285+0000 7fd557fff640 1 -- 172.21.15.16:0/3062878508 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd56400cad0 con 0x7fd5700fdeb0 2024-06-18T02:59:44.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:44.285+0000 7fd575cb1640 1 -- 172.21.15.16:0/3062878508 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd57010d7c0 con 0x7fd5700fdeb0 2024-06-18T02:59:44.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:44.286+0000 7fd575cb1640 1 -- 172.21.15.16:0/3062878508 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5700fda50 con 0x7fd5700fdeb0 2024-06-18T02:59:44.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:44.286+0000 7fd557fff640 1 -- 172.21.15.16:0/3062878508 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fd56400cc30 con 0x7fd5700fdeb0 2024-06-18T02:59:44.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:44.287+0000 7fd557fff640 1 --2- 172.21.15.16:0/3062878508 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fd558065430 0x7fd5580678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:59:44.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:44.287+0000 7fd557fff640 1 -- 172.21.15.16:0/3062878508 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fd564082080 con 0x7fd5700fdeb0 2024-06-18T02:59:44.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:44.287+0000 7fd56f7fe640 1 --2- 172.21.15.16:0/3062878508 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fd558065430 0x7fd5580678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:59:44.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:44.287+0000 7fd56f7fe640 1 --2- 172.21.15.16:0/3062878508 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fd558065430 0x7fd5580678e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd5600048a0 tx=0x7fd560035040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:59:44.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:44.291+0000 7fd557fff640 1 -- 172.21.15.16:0/3062878508 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fd564050600 con 0x7fd5700fdeb0 2024-06-18T02:59:44.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:44.500+0000 7fd575cb1640 1 -- 172.21.15.16:0/3062878508 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd57010e690 con 0x7fd5700fdeb0 2024-06-18T02:59:44.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:44.500+0000 7fd557fff640 1 -- 172.21.15.16:0/3062878508 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fd564053c20 con 0x7fd5700fdeb0 2024-06-18T02:59:44.502 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:59:44.502 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-18T02:59:44.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:44.502+0000 7fd575cb1640 1 -- 172.21.15.16:0/3062878508 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fd558065430 msgr2=0x7fd5580678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:59:44.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:44.502+0000 7fd575cb1640 1 --2- 172.21.15.16:0/3062878508 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fd558065430 0x7fd5580678e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd5600048a0 tx=0x7fd560035040 comp rx=0 tx=0).stop 2024-06-18T02:59:44.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:44.502+0000 7fd575cb1640 1 -- 172.21.15.16:0/3062878508 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5700fdeb0 msgr2=0x7fd570113560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:59:44.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:44.502+0000 7fd575cb1640 1 --2- 172.21.15.16:0/3062878508 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5700fdeb0 0x7fd570113560 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fd56400eb70 tx=0x7fd56400b040 comp rx=0 tx=0).stop 2024-06-18T02:59:44.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:44.503+0000 7fd575cb1640 1 -- 172.21.15.16:0/3062878508 shutdown_connections 2024-06-18T02:59:44.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:44.503+0000 7fd575cb1640 1 --2- 172.21.15.16:0/3062878508 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fd558065430 0x7fd5580678e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:59:44.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:44.503+0000 7fd575cb1640 1 --2- 172.21.15.16:0/3062878508 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5700fdeb0 0x7fd570113560 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:59:44.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:44.503+0000 7fd575cb1640 1 --2- 172.21.15.16:0/3062878508 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd570065010 0x7fd570113020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:59:44.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:44.503+0000 7fd575cb1640 1 -- 172.21.15.16:0/3062878508 >> 172.21.15.16:0/3062878508 conn(0x7fd570072110 msgr2=0x7fd5700f9f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:59:44.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:44.503+0000 7fd575cb1640 1 -- 172.21.15.16:0/3062878508 shutdown_connections 2024-06-18T02:59:44.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:44.503+0000 7fd575cb1640 1 -- 172.21.15.16:0/3062878508 wait complete. 2024-06-18T02:59:44.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:59:44 smithi123 ceph-mon[36358]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:59:44.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:59:44 smithi016 ceph-mon[29214]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:59:45.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:59:45 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/3062878508' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:59:45.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:59:45 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/3062878508' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:59:46.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:59:46 smithi123 ceph-mon[36358]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:59:46.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:59:46 smithi016 ceph-mon[29214]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:59:48.158 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T02:59:48.438 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T02:59:48.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:59:48 smithi123 ceph-mon[36358]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:59:48.792 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:59:48 smithi016 ceph-mon[29214]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:59:50.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:50.243+0000 7f04c9983640 1 -- 172.21.15.16:0/2681360646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04c41009f0 msgr2=0x7f04c4100dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:59:50.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:50.243+0000 7f04c9983640 1 --2- 172.21.15.16:0/2681360646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04c41009f0 0x7f04c4100dc0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f04b40049c0 tx=0x7f04b400aeb0 comp rx=0 tx=0).stop 2024-06-18T02:59:50.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:50.243+0000 7f04c9983640 1 -- 172.21.15.16:0/2681360646 shutdown_connections 2024-06-18T02:59:50.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:50.243+0000 7f04c9983640 1 --2- 172.21.15.16:0/2681360646 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f04c4101300 0x7f04c40fa9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:59:50.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:50.243+0000 7f04c9983640 1 --2- 172.21.15.16:0/2681360646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04c41009f0 0x7f04c4100dc0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:59:50.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:50.243+0000 7f04c9983640 1 -- 172.21.15.16:0/2681360646 >> 172.21.15.16:0/2681360646 conn(0x7f04c4070a40 msgr2=0x7f04c4070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:59:50.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:50.243+0000 7f04c9983640 1 -- 172.21.15.16:0/2681360646 shutdown_connections 2024-06-18T02:59:50.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:50.243+0000 7f04c9983640 1 -- 172.21.15.16:0/2681360646 wait complete. 2024-06-18T02:59:50.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:50.244+0000 7f04c9983640 1 Processor -- start 2024-06-18T02:59:50.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:50.244+0000 7f04c9983640 1 -- start start 2024-06-18T02:59:50.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:50.244+0000 7f04c9983640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04c41009f0 0x7f04c4110eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:59:50.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:50.244+0000 7f04c9983640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f04c4101300 0x7f04c41113f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:59:50.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:50.244+0000 7f04c9983640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04c41119f0 con 0x7f04c41009f0 2024-06-18T02:59:50.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:50.244+0000 7f04c9983640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04c410ae30 con 0x7f04c4101300 2024-06-18T02:59:50.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:50.244+0000 7f04c2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04c41009f0 0x7f04c4110eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:59:50.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:50.244+0000 7f04c2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04c41009f0 0x7f04c4110eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33528/0 (socket says 172.21.15.16:33528) 2024-06-18T02:59:50.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:50.244+0000 7f04c2ffd640 1 -- 172.21.15.16:0/1090562205 learned_addr learned my addr 172.21.15.16:0/1090562205 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:59:50.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:50.244+0000 7f04c27fc640 1 --2- 172.21.15.16:0/1090562205 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f04c4101300 0x7f04c41113f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:59:50.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:50.244+0000 7f04c2ffd640 1 -- 172.21.15.16:0/1090562205 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f04c4101300 msgr2=0x7f04c41113f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:59:50.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:50.244+0000 7f04c2ffd640 1 --2- 172.21.15.16:0/1090562205 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f04c4101300 0x7f04c41113f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:59:50.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:50.244+0000 7f04c2ffd640 1 -- 172.21.15.16:0/1090562205 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04b4005040 con 0x7f04c41009f0 2024-06-18T02:59:50.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:50.245+0000 7f04c27fc640 1 --2- 172.21.15.16:0/1090562205 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f04c4101300 0x7f04c41113f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-18T02:59:50.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:50.245+0000 7f04c2ffd640 1 --2- 172.21.15.16:0/1090562205 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04c41009f0 0x7f04c4110eb0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f04b400abd0 tx=0x7f04b400aeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:59:50.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:50.245+0000 7f04ab7fe640 1 -- 172.21.15.16:0/1090562205 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f04b4039070 con 0x7f04c41009f0 2024-06-18T02:59:50.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:50.245+0000 7f04ab7fe640 1 -- 172.21.15.16:0/1090562205 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f04b400a7b0 con 0x7f04c41009f0 2024-06-18T02:59:50.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:50.245+0000 7f04ab7fe640 1 -- 172.21.15.16:0/1090562205 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f04b403ec50 con 0x7f04c41009f0 2024-06-18T02:59:50.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:50.245+0000 7f04c9983640 1 -- 172.21.15.16:0/1090562205 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04c410b0b0 con 0x7f04c41009f0 2024-06-18T02:59:50.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:50.245+0000 7f04c9983640 1 -- 172.21.15.16:0/1090562205 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04c410b600 con 0x7f04c41009f0 2024-06-18T02:59:50.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:50.246+0000 7f04c9983640 1 -- 172.21.15.16:0/1090562205 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04c40fc0d0 con 0x7f04c41009f0 2024-06-18T02:59:50.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:50.246+0000 7f04ab7fe640 1 -- 172.21.15.16:0/1090562205 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f04b4031020 con 0x7f04c41009f0 2024-06-18T02:59:50.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:50.247+0000 7f04ab7fe640 1 --2- 172.21.15.16:0/1090562205 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f04ac065430 0x7f04ac0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:59:50.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:50.247+0000 7f04c27fc640 1 --2- 172.21.15.16:0/1090562205 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f04ac065430 0x7f04ac0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:59:50.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:50.247+0000 7f04ab7fe640 1 -- 172.21.15.16:0/1090562205 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f04b40a3800 con 0x7f04c41009f0 2024-06-18T02:59:50.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:50.248+0000 7f04c27fc640 1 --2- 172.21.15.16:0/1090562205 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f04ac065430 0x7f04ac0678e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f04b8007fd0 tx=0x7f04b8009e90 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:59:50.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:50.251+0000 7f04ab7fe640 1 -- 172.21.15.16:0/1090562205 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f04b4071c10 con 0x7f04c41009f0 2024-06-18T02:59:50.466 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:50.466+0000 7f04c9983640 1 -- 172.21.15.16:0/1090562205 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f04c4074960 con 0x7f04c41009f0 2024-06-18T02:59:50.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:50.467+0000 7f04ab7fe640 1 -- 172.21.15.16:0/1090562205 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f04b4075230 con 0x7f04c41009f0 2024-06-18T02:59:50.468 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:59:50.468 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-18T02:59:50.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:50.469+0000 7f04c9983640 1 -- 172.21.15.16:0/1090562205 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f04ac065430 msgr2=0x7f04ac0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:59:50.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:50.469+0000 7f04c9983640 1 --2- 172.21.15.16:0/1090562205 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f04ac065430 0x7f04ac0678e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f04b8007fd0 tx=0x7f04b8009e90 comp rx=0 tx=0).stop 2024-06-18T02:59:50.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:50.469+0000 7f04c9983640 1 -- 172.21.15.16:0/1090562205 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04c41009f0 msgr2=0x7f04c4110eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:59:50.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:50.469+0000 7f04c9983640 1 --2- 172.21.15.16:0/1090562205 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04c41009f0 0x7f04c4110eb0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f04b400abd0 tx=0x7f04b400aeb0 comp rx=0 tx=0).stop 2024-06-18T02:59:50.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:50.469+0000 7f04c9983640 1 -- 172.21.15.16:0/1090562205 shutdown_connections 2024-06-18T02:59:50.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:50.469+0000 7f04c9983640 1 --2- 172.21.15.16:0/1090562205 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f04ac065430 0x7f04ac0678e0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:59:50.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:50.469+0000 7f04c9983640 1 --2- 172.21.15.16:0/1090562205 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f04c4101300 0x7f04c41113f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:59:50.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:50.469+0000 7f04c9983640 1 --2- 172.21.15.16:0/1090562205 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04c41009f0 0x7f04c4110eb0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:59:50.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:50.469+0000 7f04c9983640 1 -- 172.21.15.16:0/1090562205 >> 172.21.15.16:0/1090562205 conn(0x7f04c4070a40 msgr2=0x7f04c4103720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:59:50.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:50.470+0000 7f04c9983640 1 -- 172.21.15.16:0/1090562205 shutdown_connections 2024-06-18T02:59:50.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:50.470+0000 7f04c9983640 1 -- 172.21.15.16:0/1090562205 wait complete. 2024-06-18T02:59:50.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:59:50 smithi016 ceph-mon[29214]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:59:50.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:59:50 smithi123 ceph-mon[36358]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:59:51.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:59:51 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/1090562205' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:59:51.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:59:51 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/1090562205' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:59:52.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:59:52 smithi123 ceph-mon[36358]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:59:52.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:59:52 smithi016 ceph-mon[29214]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:59:54.186 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T02:59:54.465 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T02:59:54.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:59:54 smithi123 ceph-mon[36358]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:59:54.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:59:54 smithi016 ceph-mon[29214]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:59:56.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:56.121+0000 7f216d10a640 1 -- 172.21.15.16:0/122355083 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21680fd310 msgr2=0x7f21680fd6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:59:56.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:56.121+0000 7f216d10a640 1 --2- 172.21.15.16:0/122355083 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21680fd310 0x7f21680fd6e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f21580048d0 tx=0x7f2158009da0 comp rx=0 tx=0).stop 2024-06-18T02:59:56.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:56.121+0000 7f216d10a640 1 -- 172.21.15.16:0/122355083 shutdown_connections 2024-06-18T02:59:56.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:56.121+0000 7f216d10a640 1 --2- 172.21.15.16:0/122355083 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21680fdcb0 0x7f2168106380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:59:56.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:56.121+0000 7f216d10a640 1 --2- 172.21.15.16:0/122355083 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21680fd310 0x7f21680fd6e0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:59:56.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:56.121+0000 7f216d10a640 1 -- 172.21.15.16:0/122355083 >> 172.21.15.16:0/122355083 conn(0x7f21680f6db0 msgr2=0x7f21680f91c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:59:56.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:56.121+0000 7f216d10a640 1 -- 172.21.15.16:0/122355083 shutdown_connections 2024-06-18T02:59:56.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:56.121+0000 7f216d10a640 1 -- 172.21.15.16:0/122355083 wait complete. 2024-06-18T02:59:56.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:56.122+0000 7f216d10a640 1 Processor -- start 2024-06-18T02:59:56.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:56.122+0000 7f216d10a640 1 -- start start 2024-06-18T02:59:56.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:56.122+0000 7f216d10a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21680fd310 0x7f21680fb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:59:56.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:56.122+0000 7f216d10a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21680fdcb0 0x7f21680fb710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:59:56.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:56.122+0000 7f216d10a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21680fbc50 con 0x7f21680fdcb0 2024-06-18T02:59:56.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:56.122+0000 7f216d10a640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21680fbdc0 con 0x7f21680fd310 2024-06-18T02:59:56.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:56.122+0000 7f215ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21680fdcb0 0x7f21680fb710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:59:56.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:56.122+0000 7f215ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21680fdcb0 0x7f21680fb710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57076/0 (socket says 172.21.15.16:57076) 2024-06-18T02:59:56.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:56.122+0000 7f215ffff640 1 -- 172.21.15.16:0/192298237 learned_addr learned my addr 172.21.15.16:0/192298237 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T02:59:56.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:56.122+0000 7f2166d9d640 1 --2- 172.21.15.16:0/192298237 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21680fd310 0x7f21680fb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:59:56.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:56.122+0000 7f215ffff640 1 -- 172.21.15.16:0/192298237 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21680fd310 msgr2=0x7f21680fb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:59:56.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:56.122+0000 7f215ffff640 1 --2- 172.21.15.16:0/192298237 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21680fd310 0x7f21680fb1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:59:56.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:56.122+0000 7f215ffff640 1 -- 172.21.15.16:0/192298237 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2158005040 con 0x7f21680fdcb0 2024-06-18T02:59:56.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:56.123+0000 7f215ffff640 1 --2- 172.21.15.16:0/192298237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21680fdcb0 0x7f21680fb710 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f215000c830 tx=0x7f2150004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:59:56.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:56.123+0000 7f215f7fe640 1 -- 172.21.15.16:0/192298237 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f215000a7d0 con 0x7f21680fdcb0 2024-06-18T02:59:56.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:56.123+0000 7f215f7fe640 1 -- 172.21.15.16:0/192298237 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f215000ae00 con 0x7f21680fdcb0 2024-06-18T02:59:56.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:56.123+0000 7f216d10a640 1 -- 172.21.15.16:0/192298237 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21680fc050 con 0x7f21680fdcb0 2024-06-18T02:59:56.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:56.123+0000 7f215f7fe640 1 -- 172.21.15.16:0/192298237 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f21500113a0 con 0x7f21680fdcb0 2024-06-18T02:59:56.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:56.123+0000 7f216d10a640 1 -- 172.21.15.16:0/192298237 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2168119730 con 0x7f21680fdcb0 2024-06-18T02:59:56.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:56.124+0000 7f216d10a640 1 -- 172.21.15.16:0/192298237 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2168065ff0 con 0x7f21680fdcb0 2024-06-18T02:59:56.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:56.124+0000 7f215f7fe640 1 -- 172.21.15.16:0/192298237 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f2150012400 con 0x7f21680fdcb0 2024-06-18T02:59:56.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:56.125+0000 7f215f7fe640 1 --2- 172.21.15.16:0/192298237 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f2154065430 0x7f21540678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T02:59:56.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:56.125+0000 7f215f7fe640 1 -- 172.21.15.16:0/192298237 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f21500824e0 con 0x7f21680fdcb0 2024-06-18T02:59:56.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:56.125+0000 7f2166d9d640 1 --2- 172.21.15.16:0/192298237 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f2154065430 0x7f21540678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T02:59:56.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:56.125+0000 7f2166d9d640 1 --2- 172.21.15.16:0/192298237 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f2154065430 0x7f21540678e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f2158000c00 tx=0x7f2158035040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T02:59:56.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:56.129+0000 7f215f7fe640 1 -- 172.21.15.16:0/192298237 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f2150050ab0 con 0x7f21680fdcb0 2024-06-18T02:59:56.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:56.341+0000 7f216d10a640 1 -- 172.21.15.16:0/192298237 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f21680fc8c0 con 0x7f21680fdcb0 2024-06-18T02:59:56.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:56.341+0000 7f215f7fe640 1 -- 172.21.15.16:0/192298237 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f21500540d0 con 0x7f21680fdcb0 2024-06-18T02:59:56.342 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T02:59:56.342 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-18T02:59:56.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:56.343+0000 7f216d10a640 1 -- 172.21.15.16:0/192298237 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f2154065430 msgr2=0x7f21540678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:59:56.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:56.343+0000 7f216d10a640 1 --2- 172.21.15.16:0/192298237 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f2154065430 0x7f21540678e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f2158000c00 tx=0x7f2158035040 comp rx=0 tx=0).stop 2024-06-18T02:59:56.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:56.343+0000 7f216d10a640 1 -- 172.21.15.16:0/192298237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21680fdcb0 msgr2=0x7f21680fb710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T02:59:56.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:56.343+0000 7f216d10a640 1 --2- 172.21.15.16:0/192298237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21680fdcb0 0x7f21680fb710 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f215000c830 tx=0x7f2150004430 comp rx=0 tx=0).stop 2024-06-18T02:59:56.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:56.344+0000 7f216d10a640 1 -- 172.21.15.16:0/192298237 shutdown_connections 2024-06-18T02:59:56.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:56.344+0000 7f216d10a640 1 --2- 172.21.15.16:0/192298237 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f2154065430 0x7f21540678e0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:59:56.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:56.344+0000 7f216d10a640 1 --2- 172.21.15.16:0/192298237 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21680fdcb0 0x7f21680fb710 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:59:56.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:56.344+0000 7f216d10a640 1 --2- 172.21.15.16:0/192298237 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f21680fd310 0x7f21680fb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T02:59:56.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:56.344+0000 7f216d10a640 1 -- 172.21.15.16:0/192298237 >> 172.21.15.16:0/192298237 conn(0x7f21680f6db0 msgr2=0x7f2168100c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T02:59:56.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:56.344+0000 7f216d10a640 1 -- 172.21.15.16:0/192298237 shutdown_connections 2024-06-18T02:59:56.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T02:59:56.344+0000 7f216d10a640 1 -- 172.21.15.16:0/192298237 wait complete. 2024-06-18T02:59:56.646 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:59:56 smithi016 ceph-mon[29214]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:59:56.646 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:59:56 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:59:56.646 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:59:56 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/192298237' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:59:56.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:59:56 smithi123 ceph-mon[36358]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:59:56.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:59:56 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T02:59:56.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:59:56 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/192298237' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T02:59:58.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 02:59:58 smithi123 ceph-mon[36358]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T02:59:58.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 02:59:58 smithi016 ceph-mon[29214]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:00:00.033 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T03:00:00.316 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T03:00:00.575 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:00:00 smithi016 ceph-mon[29214]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:00:00.576 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:00:00 smithi016 ceph-mon[29214]: overall HEALTH_WARN 2 mgr modules have recently crashed 2024-06-18T03:00:00.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:00:00 smithi123 ceph-mon[36358]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:00:00.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:00:00 smithi123 ceph-mon[36358]: overall HEALTH_WARN 2 mgr modules have recently crashed 2024-06-18T03:00:02.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:02.049+0000 7fe7235a5640 1 -- 172.21.15.16:0/3454285693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe71c102c00 msgr2=0x7fe71c0fc290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:00:02.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:02.049+0000 7fe7235a5640 1 --2- 172.21.15.16:0/3454285693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe71c102c00 0x7fe71c0fc290 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fe7100048d0 tx=0x7fe710009da0 comp rx=0 tx=0).stop 2024-06-18T03:00:02.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:02.049+0000 7fe7235a5640 1 -- 172.21.15.16:0/3454285693 shutdown_connections 2024-06-18T03:00:02.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:02.049+0000 7fe7235a5640 1 --2- 172.21.15.16:0/3454285693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe71c102c00 0x7fe71c0fc290 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:00:02.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:02.049+0000 7fe7235a5640 1 --2- 172.21.15.16:0/3454285693 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe71c102260 0x7fe71c102630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:00:02.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:02.049+0000 7fe7235a5640 1 -- 172.21.15.16:0/3454285693 >> 172.21.15.16:0/3454285693 conn(0x7fe71c0f7e50 msgr2=0x7fe71c0fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:00:02.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:02.049+0000 7fe7235a5640 1 -- 172.21.15.16:0/3454285693 shutdown_connections 2024-06-18T03:00:02.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:02.049+0000 7fe7235a5640 1 -- 172.21.15.16:0/3454285693 wait complete. 2024-06-18T03:00:02.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:02.050+0000 7fe7235a5640 1 Processor -- start 2024-06-18T03:00:02.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:02.050+0000 7fe7235a5640 1 -- start start 2024-06-18T03:00:02.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:02.050+0000 7fe7235a5640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe71c102260 0x7fe71c06f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:00:02.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:02.050+0000 7fe7235a5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe71c102c00 0x7fe71c06fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:00:02.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:02.050+0000 7fe7235a5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe71c0701f0 con 0x7fe71c102c00 2024-06-18T03:00:02.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:02.050+0000 7fe7235a5640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe71c070360 con 0x7fe71c102260 2024-06-18T03:00:02.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:02.050+0000 7fe720b40640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe71c102c00 0x7fe71c06fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:00:02.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:02.050+0000 7fe720b40640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe71c102c00 0x7fe71c06fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39472/0 (socket says 172.21.15.16:39472) 2024-06-18T03:00:02.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:02.050+0000 7fe720b40640 1 -- 172.21.15.16:0/3653514129 learned_addr learned my addr 172.21.15.16:0/3653514129 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T03:00:02.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:02.050+0000 7fe721341640 1 --2- 172.21.15.16:0/3653514129 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe71c102260 0x7fe71c06f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:00:02.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:02.050+0000 7fe720b40640 1 -- 172.21.15.16:0/3653514129 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe71c102260 msgr2=0x7fe71c06f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:00:02.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:02.050+0000 7fe720b40640 1 --2- 172.21.15.16:0/3653514129 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe71c102260 0x7fe71c06f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:00:02.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:02.050+0000 7fe720b40640 1 -- 172.21.15.16:0/3653514129 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe710005040 con 0x7fe71c102c00 2024-06-18T03:00:02.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:02.050+0000 7fe721341640 1 --2- 172.21.15.16:0/3653514129 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe71c102260 0x7fe71c06f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-18T03:00:02.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:02.050+0000 7fe720b40640 1 --2- 172.21.15.16:0/3653514129 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe71c102c00 0x7fe71c06fcb0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fe7100043d0 tx=0x7fe710009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:00:02.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:02.051+0000 7fe719ffb640 1 -- 172.21.15.16:0/3653514129 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe710038070 con 0x7fe71c102c00 2024-06-18T03:00:02.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:02.051+0000 7fe719ffb640 1 -- 172.21.15.16:0/3653514129 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe710032970 con 0x7fe71c102c00 2024-06-18T03:00:02.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:02.051+0000 7fe7235a5640 1 -- 172.21.15.16:0/3653514129 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe71c070590 con 0x7fe71c102c00 2024-06-18T03:00:02.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:02.051+0000 7fe719ffb640 1 -- 172.21.15.16:0/3653514129 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe71003da60 con 0x7fe71c102c00 2024-06-18T03:00:02.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:02.051+0000 7fe7235a5640 1 -- 172.21.15.16:0/3653514129 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe71c119640 con 0x7fe71c102c00 2024-06-18T03:00:02.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:02.052+0000 7fe7235a5640 1 -- 172.21.15.16:0/3653514129 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe71c0fd9d0 con 0x7fe71c102c00 2024-06-18T03:00:02.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:02.052+0000 7fe719ffb640 1 -- 172.21.15.16:0/3653514129 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fe710030020 con 0x7fe71c102c00 2024-06-18T03:00:02.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:02.053+0000 7fe719ffb640 1 --2- 172.21.15.16:0/3653514129 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fe704065430 0x7fe7040678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:00:02.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:02.053+0000 7fe719ffb640 1 -- 172.21.15.16:0/3653514129 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fe7100a25d0 con 0x7fe71c102c00 2024-06-18T03:00:02.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:02.053+0000 7fe721341640 1 --2- 172.21.15.16:0/3653514129 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fe704065430 0x7fe7040678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:00:02.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:02.054+0000 7fe721341640 1 --2- 172.21.15.16:0/3653514129 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fe704065430 0x7fe7040678e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fe70c007ee0 tx=0x7fe70c00b040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:00:02.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:02.057+0000 7fe719ffb640 1 -- 172.21.15.16:0/3653514129 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fe710070a80 con 0x7fe71c102c00 2024-06-18T03:00:02.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:02.268+0000 7fe7235a5640 1 -- 172.21.15.16:0/3653514129 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe71c05ead0 con 0x7fe71c102c00 2024-06-18T03:00:02.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:02.269+0000 7fe719ffb640 1 -- 172.21.15.16:0/3653514129 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fe7100740a0 con 0x7fe71c102c00 2024-06-18T03:00:02.269 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T03:00:02.269 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-18T03:00:02.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:02.271+0000 7fe7235a5640 1 -- 172.21.15.16:0/3653514129 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fe704065430 msgr2=0x7fe7040678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:00:02.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:02.271+0000 7fe7235a5640 1 --2- 172.21.15.16:0/3653514129 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fe704065430 0x7fe7040678e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fe70c007ee0 tx=0x7fe70c00b040 comp rx=0 tx=0).stop 2024-06-18T03:00:02.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:02.271+0000 7fe7235a5640 1 -- 172.21.15.16:0/3653514129 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe71c102c00 msgr2=0x7fe71c06fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:00:02.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:02.271+0000 7fe7235a5640 1 --2- 172.21.15.16:0/3653514129 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe71c102c00 0x7fe71c06fcb0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fe7100043d0 tx=0x7fe710009da0 comp rx=0 tx=0).stop 2024-06-18T03:00:02.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:02.271+0000 7fe7235a5640 1 -- 172.21.15.16:0/3653514129 shutdown_connections 2024-06-18T03:00:02.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:02.271+0000 7fe7235a5640 1 --2- 172.21.15.16:0/3653514129 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fe704065430 0x7fe7040678e0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:00:02.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:02.271+0000 7fe7235a5640 1 --2- 172.21.15.16:0/3653514129 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe71c102c00 0x7fe71c06fcb0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:00:02.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:02.271+0000 7fe7235a5640 1 --2- 172.21.15.16:0/3653514129 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe71c102260 0x7fe71c06f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:00:02.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:02.271+0000 7fe7235a5640 1 -- 172.21.15.16:0/3653514129 >> 172.21.15.16:0/3653514129 conn(0x7fe71c0f7e50 msgr2=0x7fe71c0f8e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:00:02.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:02.271+0000 7fe7235a5640 1 -- 172.21.15.16:0/3653514129 shutdown_connections 2024-06-18T03:00:02.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:02.271+0000 7fe7235a5640 1 -- 172.21.15.16:0/3653514129 wait complete. 2024-06-18T03:00:02.886 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:00:02 smithi016 ceph-mon[29214]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:00:02.886 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:00:02 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/3653514129' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:00:02.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:00:02 smithi123 ceph-mon[36358]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:00:02.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:00:02 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/3653514129' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:00:04.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:00:04 smithi123 ceph-mon[36358]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:00:05.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:00:04 smithi016 ceph-mon[29214]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:00:05.917 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T03:00:06.194 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T03:00:06.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:00:06 smithi123 ceph-mon[36358]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:00:07.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:00:06 smithi016 ceph-mon[29214]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:00:07.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:07.910+0000 7f9a560ea640 1 -- 172.21.15.16:0/3022642977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a500652f0 msgr2=0x7f9a50100f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:00:07.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:07.910+0000 7f9a560ea640 1 --2- 172.21.15.16:0/3022642977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a500652f0 0x7f9a50100f30 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f9a440048d0 tx=0x7f9a44009da0 comp rx=0 tx=0).stop 2024-06-18T03:00:07.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:07.910+0000 7f9a560ea640 1 -- 172.21.15.16:0/3022642977 shutdown_connections 2024-06-18T03:00:07.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:07.911+0000 7f9a560ea640 1 --2- 172.21.15.16:0/3022642977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a500652f0 0x7f9a50100f30 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:00:07.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:07.911+0000 7f9a560ea640 1 --2- 172.21.15.16:0/3022642977 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a50064f20 0x7f9a501009f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:00:07.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:07.911+0000 7f9a560ea640 1 -- 172.21.15.16:0/3022642977 >> 172.21.15.16:0/3022642977 conn(0x7f9a50070a40 msgr2=0x7f9a50070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:00:07.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:07.911+0000 7f9a560ea640 1 -- 172.21.15.16:0/3022642977 shutdown_connections 2024-06-18T03:00:07.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:07.911+0000 7f9a560ea640 1 -- 172.21.15.16:0/3022642977 wait complete. 2024-06-18T03:00:07.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:07.911+0000 7f9a560ea640 1 Processor -- start 2024-06-18T03:00:07.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:07.911+0000 7f9a560ea640 1 -- start start 2024-06-18T03:00:07.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:07.912+0000 7f9a560ea640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a50064f20 0x7f9a50112fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:00:07.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:07.912+0000 7f9a560ea640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a500652f0 0x7f9a50113520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:00:07.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:07.912+0000 7f9a560ea640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a50113b20 con 0x7f9a500652f0 2024-06-18T03:00:07.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:07.912+0000 7f9a560ea640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a50113c90 con 0x7f9a50064f20 2024-06-18T03:00:07.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:07.912+0000 7f9a4effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a500652f0 0x7f9a50113520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:00:07.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:07.912+0000 7f9a4effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a500652f0 0x7f9a50113520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:39490/0 (socket says 172.21.15.16:39490) 2024-06-18T03:00:07.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:07.912+0000 7f9a4effd640 1 -- 172.21.15.16:0/2021390372 learned_addr learned my addr 172.21.15.16:0/2021390372 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T03:00:07.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:07.912+0000 7f9a4f7fe640 1 --2- 172.21.15.16:0/2021390372 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a50064f20 0x7f9a50112fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:00:07.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:07.912+0000 7f9a4effd640 1 -- 172.21.15.16:0/2021390372 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a50064f20 msgr2=0x7f9a50112fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:00:07.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:07.912+0000 7f9a4effd640 1 --2- 172.21.15.16:0/2021390372 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a50064f20 0x7f9a50112fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:00:07.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:07.912+0000 7f9a4effd640 1 -- 172.21.15.16:0/2021390372 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a44005040 con 0x7f9a500652f0 2024-06-18T03:00:07.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:07.913+0000 7f9a4effd640 1 --2- 172.21.15.16:0/2021390372 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a500652f0 0x7f9a50113520 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f9a44004930 tx=0x7f9a44009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:00:07.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:07.913+0000 7f9a37fff640 1 -- 172.21.15.16:0/2021390372 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9a44038070 con 0x7f9a500652f0 2024-06-18T03:00:07.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:07.913+0000 7f9a37fff640 1 -- 172.21.15.16:0/2021390372 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9a44032d60 con 0x7f9a500652f0 2024-06-18T03:00:07.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:07.913+0000 7f9a560ea640 1 -- 172.21.15.16:0/2021390372 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a5010d0b0 con 0x7f9a500652f0 2024-06-18T03:00:07.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:07.913+0000 7f9a37fff640 1 -- 172.21.15.16:0/2021390372 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9a4403ddb0 con 0x7f9a500652f0 2024-06-18T03:00:07.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:07.913+0000 7f9a560ea640 1 -- 172.21.15.16:0/2021390372 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a5010d5f0 con 0x7f9a500652f0 2024-06-18T03:00:07.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:07.914+0000 7f9a560ea640 1 -- 172.21.15.16:0/2021390372 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a500fc0d0 con 0x7f9a500652f0 2024-06-18T03:00:07.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:07.915+0000 7f9a37fff640 1 -- 172.21.15.16:0/2021390372 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f9a44030020 con 0x7f9a500652f0 2024-06-18T03:00:07.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:07.915+0000 7f9a37fff640 1 --2- 172.21.15.16:0/2021390372 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f9a38065430 0x7f9a380678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:00:07.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:07.915+0000 7f9a37fff640 1 -- 172.21.15.16:0/2021390372 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f9a440a2830 con 0x7f9a500652f0 2024-06-18T03:00:07.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:07.915+0000 7f9a4f7fe640 1 --2- 172.21.15.16:0/2021390372 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f9a38065430 0x7f9a380678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:00:07.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:07.916+0000 7f9a4f7fe640 1 --2- 172.21.15.16:0/2021390372 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f9a38065430 0x7f9a380678e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f9a40004eb0 tx=0x7f9a4000b040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:00:07.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:07.920+0000 7f9a37fff640 1 -- 172.21.15.16:0/2021390372 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f9a44070d60 con 0x7f9a500652f0 2024-06-18T03:00:08.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:08.131+0000 7f9a560ea640 1 -- 172.21.15.16:0/2021390372 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f9a50108360 con 0x7f9a500652f0 2024-06-18T03:00:08.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:08.131+0000 7f9a37fff640 1 -- 172.21.15.16:0/2021390372 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f9a44074380 con 0x7f9a500652f0 2024-06-18T03:00:08.132 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T03:00:08.132 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-18T03:00:08.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:08.133+0000 7f9a560ea640 1 -- 172.21.15.16:0/2021390372 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f9a38065430 msgr2=0x7f9a380678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:00:08.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:08.133+0000 7f9a560ea640 1 --2- 172.21.15.16:0/2021390372 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f9a38065430 0x7f9a380678e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f9a40004eb0 tx=0x7f9a4000b040 comp rx=0 tx=0).stop 2024-06-18T03:00:08.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:08.133+0000 7f9a560ea640 1 -- 172.21.15.16:0/2021390372 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a500652f0 msgr2=0x7f9a50113520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:00:08.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:08.133+0000 7f9a560ea640 1 --2- 172.21.15.16:0/2021390372 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a500652f0 0x7f9a50113520 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f9a44004930 tx=0x7f9a44009da0 comp rx=0 tx=0).stop 2024-06-18T03:00:08.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:08.133+0000 7f9a560ea640 1 -- 172.21.15.16:0/2021390372 shutdown_connections 2024-06-18T03:00:08.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:08.133+0000 7f9a560ea640 1 --2- 172.21.15.16:0/2021390372 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f9a38065430 0x7f9a380678e0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:00:08.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:08.133+0000 7f9a560ea640 1 --2- 172.21.15.16:0/2021390372 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a500652f0 0x7f9a50113520 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:00:08.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:08.133+0000 7f9a560ea640 1 --2- 172.21.15.16:0/2021390372 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a50064f20 0x7f9a50112fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:00:08.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:08.133+0000 7f9a560ea640 1 -- 172.21.15.16:0/2021390372 >> 172.21.15.16:0/2021390372 conn(0x7f9a50070a40 msgr2=0x7f9a50101770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:00:08.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:08.133+0000 7f9a560ea640 1 -- 172.21.15.16:0/2021390372 shutdown_connections 2024-06-18T03:00:08.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:08.133+0000 7f9a560ea640 1 -- 172.21.15.16:0/2021390372 wait complete. 2024-06-18T03:00:08.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:00:08 smithi016 ceph-mon[29214]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:00:08.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:00:08 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/2021390372' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:00:08.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:00:08 smithi123 ceph-mon[36358]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:00:08.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:00:08 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/2021390372' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:00:10.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:00:10 smithi123 ceph-mon[36358]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:00:11.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:00:10 smithi016 ceph-mon[29214]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:00:11.929 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T03:00:11.958 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:00:11 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T03:00:11.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:00:11 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T03:00:12.207 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T03:00:12.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:00:12 smithi123 ceph-mon[36358]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:00:13.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:00:12 smithi016 ceph-mon[29214]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:00:13.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:13.966+0000 7fb379a60640 1 -- 172.21.15.16:0/935866258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb374070120 msgr2=0x7fb3741085d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:00:13.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:13.966+0000 7fb379a60640 1 --2- 172.21.15.16:0/935866258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb374070120 0x7fb3741085d0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fb35c0048d0 tx=0x7fb35c009da0 comp rx=0 tx=0).stop 2024-06-18T03:00:13.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:13.966+0000 7fb379a60640 1 -- 172.21.15.16:0/935866258 shutdown_connections 2024-06-18T03:00:13.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:13.966+0000 7fb379a60640 1 --2- 172.21.15.16:0/935866258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb374070120 0x7fb3741085d0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:00:13.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:13.966+0000 7fb379a60640 1 --2- 172.21.15.16:0/935866258 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb37406f810 0x7fb37406fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:00:13.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:13.966+0000 7fb379a60640 1 -- 172.21.15.16:0/935866258 >> 172.21.15.16:0/935866258 conn(0x7fb3740f7e50 msgr2=0x7fb3740fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:00:13.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:13.967+0000 7fb379a60640 1 -- 172.21.15.16:0/935866258 shutdown_connections 2024-06-18T03:00:13.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:13.967+0000 7fb379a60640 1 -- 172.21.15.16:0/935866258 wait complete. 2024-06-18T03:00:13.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:13.967+0000 7fb379a60640 1 Processor -- start 2024-06-18T03:00:13.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:13.967+0000 7fb379a60640 1 -- start start 2024-06-18T03:00:13.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:13.967+0000 7fb379a60640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb37406f810 0x7fb374115280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:00:13.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:13.967+0000 7fb379a60640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb374070120 0x7fb3741157c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:00:13.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:13.967+0000 7fb379a60640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb374115dc0 con 0x7fb37406f810 2024-06-18T03:00:13.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:13.967+0000 7fb379a60640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb37410f200 con 0x7fb374070120 2024-06-18T03:00:13.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:13.968+0000 7fb372ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb37406f810 0x7fb374115280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:00:13.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:13.968+0000 7fb372ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb37406f810 0x7fb374115280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59748/0 (socket says 172.21.15.16:59748) 2024-06-18T03:00:13.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:13.968+0000 7fb372ffd640 1 -- 172.21.15.16:0/1780426267 learned_addr learned my addr 172.21.15.16:0/1780426267 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T03:00:13.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:13.968+0000 7fb36bfff640 1 --2- 172.21.15.16:0/1780426267 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb374070120 0x7fb3741157c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:00:13.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:13.968+0000 7fb372ffd640 1 -- 172.21.15.16:0/1780426267 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb374070120 msgr2=0x7fb3741157c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:00:13.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:13.968+0000 7fb372ffd640 1 --2- 172.21.15.16:0/1780426267 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb374070120 0x7fb3741157c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:00:13.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:13.968+0000 7fb372ffd640 1 -- 172.21.15.16:0/1780426267 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb35c005040 con 0x7fb37406f810 2024-06-18T03:00:13.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:13.968+0000 7fb372ffd640 1 --2- 172.21.15.16:0/1780426267 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb37406f810 0x7fb374115280 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fb364002a50 tx=0x7fb364004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:00:13.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:13.968+0000 7fb36b7fe640 1 -- 172.21.15.16:0/1780426267 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb364007c30 con 0x7fb37406f810 2024-06-18T03:00:13.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:13.968+0000 7fb379a60640 1 -- 172.21.15.16:0/1780426267 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb37410f4e0 con 0x7fb37406f810 2024-06-18T03:00:13.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:13.968+0000 7fb36b7fe640 1 -- 172.21.15.16:0/1780426267 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb36400f040 con 0x7fb37406f810 2024-06-18T03:00:13.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:13.968+0000 7fb36b7fe640 1 -- 172.21.15.16:0/1780426267 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb36400dad0 con 0x7fb37406f810 2024-06-18T03:00:13.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:13.968+0000 7fb379a60640 1 -- 172.21.15.16:0/1780426267 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb37410f9d0 con 0x7fb37406f810 2024-06-18T03:00:13.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:13.969+0000 7fb379a60640 1 -- 172.21.15.16:0/1780426267 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb374070f50 con 0x7fb37406f810 2024-06-18T03:00:13.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:13.970+0000 7fb36b7fe640 1 -- 172.21.15.16:0/1780426267 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fb36400dd40 con 0x7fb37406f810 2024-06-18T03:00:13.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:13.970+0000 7fb36b7fe640 1 --2- 172.21.15.16:0/1780426267 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fb360065430 0x7fb3600678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:00:13.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:13.970+0000 7fb36b7fe640 1 -- 172.21.15.16:0/1780426267 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fb364081d80 con 0x7fb37406f810 2024-06-18T03:00:13.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:13.970+0000 7fb36bfff640 1 --2- 172.21.15.16:0/1780426267 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fb360065430 0x7fb3600678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:00:13.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:13.971+0000 7fb36bfff640 1 --2- 172.21.15.16:0/1780426267 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fb360065430 0x7fb3600678e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb3741009b0 tx=0x7fb35c009da0 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:00:13.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:13.974+0000 7fb36b7fe640 1 -- 172.21.15.16:0/1780426267 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fb364050230 con 0x7fb37406f810 2024-06-18T03:00:14.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:14.189+0000 7fb379a60640 1 -- 172.21.15.16:0/1780426267 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fb37404b480 con 0x7fb37406f810 2024-06-18T03:00:14.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:14.189+0000 7fb36b7fe640 1 -- 172.21.15.16:0/1780426267 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fb364053850 con 0x7fb37406f810 2024-06-18T03:00:14.190 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T03:00:14.190 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-18T03:00:14.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:14.191+0000 7fb379a60640 1 -- 172.21.15.16:0/1780426267 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fb360065430 msgr2=0x7fb3600678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:00:14.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:14.191+0000 7fb379a60640 1 --2- 172.21.15.16:0/1780426267 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fb360065430 0x7fb3600678e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb3741009b0 tx=0x7fb35c009da0 comp rx=0 tx=0).stop 2024-06-18T03:00:14.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:14.191+0000 7fb379a60640 1 -- 172.21.15.16:0/1780426267 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb37406f810 msgr2=0x7fb374115280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:00:14.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:14.191+0000 7fb379a60640 1 --2- 172.21.15.16:0/1780426267 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb37406f810 0x7fb374115280 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fb364002a50 tx=0x7fb364004600 comp rx=0 tx=0).stop 2024-06-18T03:00:14.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:14.191+0000 7fb379a60640 1 -- 172.21.15.16:0/1780426267 shutdown_connections 2024-06-18T03:00:14.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:14.191+0000 7fb379a60640 1 --2- 172.21.15.16:0/1780426267 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fb360065430 0x7fb3600678e0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:00:14.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:14.191+0000 7fb379a60640 1 --2- 172.21.15.16:0/1780426267 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb374070120 0x7fb3741157c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:00:14.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:14.191+0000 7fb379a60640 1 --2- 172.21.15.16:0/1780426267 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb37406f810 0x7fb374115280 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:00:14.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:14.191+0000 7fb379a60640 1 -- 172.21.15.16:0/1780426267 >> 172.21.15.16:0/1780426267 conn(0x7fb3740f7e50 msgr2=0x7fb3740f92c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:00:14.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:14.191+0000 7fb379a60640 1 -- 172.21.15.16:0/1780426267 shutdown_connections 2024-06-18T03:00:14.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:14.191+0000 7fb379a60640 1 -- 172.21.15.16:0/1780426267 wait complete. 2024-06-18T03:00:14.950 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:00:14 smithi016 ceph-mon[29214]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:00:14.950 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:00:14 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/1780426267' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:00:14.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:00:14 smithi123 ceph-mon[36358]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:00:14.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:00:14 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/1780426267' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:00:15.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:00:15 smithi123 ceph-mon[36358]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:00:16.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:00:15 smithi016 ceph-mon[29214]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:00:17.980 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T03:00:18.259 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T03:00:18.619 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:00:18 smithi016 ceph-mon[29214]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:00:18.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:00:18 smithi123 ceph-mon[36358]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:00:20.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:19.998+0000 7f0737a5d640 1 -- 172.21.15.16:0/2610241950 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0730071440 msgr2=0x7f073006f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:00:20.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:19.998+0000 7f0737a5d640 1 --2- 172.21.15.16:0/2610241950 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0730071440 0x7f073006f810 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f072c0048d0 tx=0x7f072c009da0 comp rx=0 tx=0).stop 2024-06-18T03:00:20.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:19.998+0000 7f0737a5d640 1 -- 172.21.15.16:0/2610241950 shutdown_connections 2024-06-18T03:00:20.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:19.998+0000 7f0737a5d640 1 --2- 172.21.15.16:0/2610241950 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f073006fd50 0x7f07300701c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:00:20.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:19.998+0000 7f0737a5d640 1 --2- 172.21.15.16:0/2610241950 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0730071440 0x7f073006f810 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:00:20.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:19.998+0000 7f0737a5d640 1 -- 172.21.15.16:0/2610241950 >> 172.21.15.16:0/2610241950 conn(0x7f07300f7e50 msgr2=0x7f07300fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:00:20.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:19.998+0000 7f0737a5d640 1 -- 172.21.15.16:0/2610241950 shutdown_connections 2024-06-18T03:00:20.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:19.998+0000 7f0737a5d640 1 -- 172.21.15.16:0/2610241950 wait complete. 2024-06-18T03:00:20.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:19.999+0000 7f0737a5d640 1 Processor -- start 2024-06-18T03:00:20.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:19.999+0000 7f0737a5d640 1 -- start start 2024-06-18T03:00:20.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:19.999+0000 7f0737a5d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f073006fd50 0x7f07301199a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:00:20.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:19.999+0000 7f0737a5d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0730071440 0x7f0730119ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:00:20.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:19.999+0000 7f0737a5d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f073011a530 con 0x7f0730071440 2024-06-18T03:00:20.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:19.999+0000 7f0737a5d640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0730113940 con 0x7f073006fd50 2024-06-18T03:00:20.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:19.999+0000 7f0734ff8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0730071440 0x7f0730119ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:00:20.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:19.999+0000 7f0734ff8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0730071440 0x7f0730119ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59762/0 (socket says 172.21.15.16:59762) 2024-06-18T03:00:20.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:19.999+0000 7f0734ff8640 1 -- 172.21.15.16:0/2797445299 learned_addr learned my addr 172.21.15.16:0/2797445299 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T03:00:20.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:19.999+0000 7f07357f9640 1 --2- 172.21.15.16:0/2797445299 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f073006fd50 0x7f07301199a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:00:20.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:19.999+0000 7f0734ff8640 1 -- 172.21.15.16:0/2797445299 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f073006fd50 msgr2=0x7f07301199a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:00:20.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:19.999+0000 7f0734ff8640 1 --2- 172.21.15.16:0/2797445299 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f073006fd50 0x7f07301199a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:00:20.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:19.999+0000 7f0734ff8640 1 -- 172.21.15.16:0/2797445299 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f072c005040 con 0x7f0730071440 2024-06-18T03:00:20.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:20.000+0000 7f0734ff8640 1 --2- 172.21.15.16:0/2797445299 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0730071440 0x7f0730119ee0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f0720002a50 tx=0x7f0720004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:00:20.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:20.000+0000 7f0725ffb640 1 -- 172.21.15.16:0/2797445299 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0720007c30 con 0x7f0730071440 2024-06-18T03:00:20.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:20.000+0000 7f0725ffb640 1 -- 172.21.15.16:0/2797445299 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f072000f040 con 0x7f0730071440 2024-06-18T03:00:20.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:20.000+0000 7f0737a5d640 1 -- 172.21.15.16:0/2797445299 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0730113c20 con 0x7f0730071440 2024-06-18T03:00:20.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:20.000+0000 7f0725ffb640 1 -- 172.21.15.16:0/2797445299 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f072000dad0 con 0x7f0730071440 2024-06-18T03:00:20.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:20.000+0000 7f0737a5d640 1 -- 172.21.15.16:0/2797445299 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0730114140 con 0x7f0730071440 2024-06-18T03:00:20.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:20.001+0000 7f0737a5d640 1 -- 172.21.15.16:0/2797445299 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0730070f50 con 0x7f0730071440 2024-06-18T03:00:20.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:20.001+0000 7f0725ffb640 1 -- 172.21.15.16:0/2797445299 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f072000dc70 con 0x7f0730071440 2024-06-18T03:00:20.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:20.002+0000 7f0725ffb640 1 --2- 172.21.15.16:0/2797445299 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f071c065430 0x7f071c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:00:20.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:20.002+0000 7f0725ffb640 1 -- 172.21.15.16:0/2797445299 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f0720081c60 con 0x7f0730071440 2024-06-18T03:00:20.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:20.002+0000 7f07357f9640 1 --2- 172.21.15.16:0/2797445299 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f071c065430 0x7f071c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:00:20.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:20.003+0000 7f07357f9640 1 --2- 172.21.15.16:0/2797445299 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f071c065430 0x7f071c0678e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f072c003080 tx=0x7f072c035040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:00:20.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:20.006+0000 7f0725ffb640 1 -- 172.21.15.16:0/2797445299 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f072004cdf0 con 0x7f0730071440 2024-06-18T03:00:20.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:20.217+0000 7f0737a5d640 1 -- 172.21.15.16:0/2797445299 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f07300fbab0 con 0x7f0730071440 2024-06-18T03:00:20.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:20.218+0000 7f0725ffb640 1 -- 172.21.15.16:0/2797445299 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f0720050190 con 0x7f0730071440 2024-06-18T03:00:20.219 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T03:00:20.219 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-18T03:00:20.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:20.220+0000 7f0737a5d640 1 -- 172.21.15.16:0/2797445299 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f071c065430 msgr2=0x7f071c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:00:20.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:20.220+0000 7f0737a5d640 1 --2- 172.21.15.16:0/2797445299 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f071c065430 0x7f071c0678e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f072c003080 tx=0x7f072c035040 comp rx=0 tx=0).stop 2024-06-18T03:00:20.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:20.220+0000 7f0737a5d640 1 -- 172.21.15.16:0/2797445299 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0730071440 msgr2=0x7f0730119ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:00:20.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:20.220+0000 7f0737a5d640 1 --2- 172.21.15.16:0/2797445299 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0730071440 0x7f0730119ee0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f0720002a50 tx=0x7f0720004600 comp rx=0 tx=0).stop 2024-06-18T03:00:20.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:20.220+0000 7f0737a5d640 1 -- 172.21.15.16:0/2797445299 shutdown_connections 2024-06-18T03:00:20.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:20.220+0000 7f0737a5d640 1 --2- 172.21.15.16:0/2797445299 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f071c065430 0x7f071c0678e0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:00:20.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:20.220+0000 7f0737a5d640 1 --2- 172.21.15.16:0/2797445299 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0730071440 0x7f0730119ee0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:00:20.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:20.220+0000 7f0737a5d640 1 --2- 172.21.15.16:0/2797445299 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f073006fd50 0x7f07301199a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:00:20.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:20.220+0000 7f0737a5d640 1 -- 172.21.15.16:0/2797445299 >> 172.21.15.16:0/2797445299 conn(0x7f07300f7e50 msgr2=0x7f07300f8f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:00:20.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:20.220+0000 7f0737a5d640 1 -- 172.21.15.16:0/2797445299 shutdown_connections 2024-06-18T03:00:20.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:20.220+0000 7f0737a5d640 1 -- 172.21.15.16:0/2797445299 wait complete. 2024-06-18T03:00:20.693 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:00:20 smithi016 ceph-mon[29214]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:00:20.693 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:00:20 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/2797445299' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:00:20.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:00:20 smithi123 ceph-mon[36358]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:00:20.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:00:20 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/2797445299' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:00:22.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:00:22 smithi123 ceph-mon[36358]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:00:22.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:00:22 smithi016 ceph-mon[29214]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:00:23.810 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T03:00:24.089 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T03:00:24.533 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:00:24 smithi016 ceph-mon[29214]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:00:24.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:00:24 smithi123 ceph-mon[36358]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:00:25.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:25.745+0000 7ff33e160640 1 -- 172.21.15.16:0/218187531 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff338102c90 msgr2=0x7ff3380fc320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:00:25.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:25.745+0000 7ff33e160640 1 --2- 172.21.15.16:0/218187531 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff338102c90 0x7ff3380fc320 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7ff32c0048d0 tx=0x7ff32c009da0 comp rx=0 tx=0).stop 2024-06-18T03:00:25.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:25.745+0000 7ff33e160640 1 -- 172.21.15.16:0/218187531 shutdown_connections 2024-06-18T03:00:25.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:25.745+0000 7ff33e160640 1 --2- 172.21.15.16:0/218187531 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff338102c90 0x7ff3380fc320 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:00:25.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:25.745+0000 7ff33e160640 1 --2- 172.21.15.16:0/218187531 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff3381022f0 0x7ff3381026c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:00:25.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:25.745+0000 7ff33e160640 1 -- 172.21.15.16:0/218187531 >> 172.21.15.16:0/218187531 conn(0x7ff3380f7f00 msgr2=0x7ff3380fa310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:00:25.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:25.745+0000 7ff33e160640 1 -- 172.21.15.16:0/218187531 shutdown_connections 2024-06-18T03:00:25.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:25.745+0000 7ff33e160640 1 -- 172.21.15.16:0/218187531 wait complete. 2024-06-18T03:00:25.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:25.745+0000 7ff33e160640 1 Processor -- start 2024-06-18T03:00:25.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:25.746+0000 7ff33e160640 1 -- start start 2024-06-18T03:00:25.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:25.746+0000 7ff33e160640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3381022f0 0x7ff338113000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:00:25.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:25.746+0000 7ff33e160640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff338102c90 0x7ff338113540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:00:25.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:25.746+0000 7ff33e160640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff338113bd0 con 0x7ff3381022f0 2024-06-18T03:00:25.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:25.746+0000 7ff33e160640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff33810cfc0 con 0x7ff338102c90 2024-06-18T03:00:25.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:25.746+0000 7ff3377fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3381022f0 0x7ff338113000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:00:25.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:25.746+0000 7ff3377fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3381022f0 0x7ff338113000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53680/0 (socket says 172.21.15.16:53680) 2024-06-18T03:00:25.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:25.746+0000 7ff3377fe640 1 -- 172.21.15.16:0/2283027579 learned_addr learned my addr 172.21.15.16:0/2283027579 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T03:00:25.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:25.746+0000 7ff336ffd640 1 --2- 172.21.15.16:0/2283027579 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff338102c90 0x7ff338113540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:00:25.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:25.746+0000 7ff3377fe640 1 -- 172.21.15.16:0/2283027579 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff338102c90 msgr2=0x7ff338113540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:00:25.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:25.746+0000 7ff3377fe640 1 --2- 172.21.15.16:0/2283027579 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff338102c90 0x7ff338113540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:00:25.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:25.746+0000 7ff3377fe640 1 -- 172.21.15.16:0/2283027579 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff32c005040 con 0x7ff3381022f0 2024-06-18T03:00:25.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:25.746+0000 7ff336ffd640 1 --2- 172.21.15.16:0/2283027579 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff338102c90 0x7ff338113540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-18T03:00:25.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:25.746+0000 7ff3377fe640 1 --2- 172.21.15.16:0/2283027579 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3381022f0 0x7ff338113000 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7ff32800b790 tx=0x7ff3280091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:00:25.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:25.747+0000 7ff31ffff640 1 -- 172.21.15.16:0/2283027579 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff32800c700 con 0x7ff3381022f0 2024-06-18T03:00:25.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:25.747+0000 7ff33e160640 1 -- 172.21.15.16:0/2283027579 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff33810d2a0 con 0x7ff3381022f0 2024-06-18T03:00:25.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:25.747+0000 7ff31ffff640 1 -- 172.21.15.16:0/2283027579 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff32800cd30 con 0x7ff3381022f0 2024-06-18T03:00:25.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:25.747+0000 7ff33e160640 1 -- 172.21.15.16:0/2283027579 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff33810d7c0 con 0x7ff3381022f0 2024-06-18T03:00:25.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:25.747+0000 7ff31ffff640 1 -- 172.21.15.16:0/2283027579 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff328013370 con 0x7ff3381022f0 2024-06-18T03:00:25.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:25.748+0000 7ff33e160640 1 -- 172.21.15.16:0/2283027579 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3380fda60 con 0x7ff3381022f0 2024-06-18T03:00:25.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:25.748+0000 7ff31ffff640 1 -- 172.21.15.16:0/2283027579 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7ff328012070 con 0x7ff3381022f0 2024-06-18T03:00:25.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:25.749+0000 7ff31ffff640 1 --2- 172.21.15.16:0/2283027579 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7ff320065430 0x7ff3200678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:00:25.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:25.749+0000 7ff31ffff640 1 -- 172.21.15.16:0/2283027579 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7ff328081f00 con 0x7ff3381022f0 2024-06-18T03:00:25.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:25.749+0000 7ff336ffd640 1 --2- 172.21.15.16:0/2283027579 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7ff320065430 0x7ff3200678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:00:25.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:25.749+0000 7ff336ffd640 1 --2- 172.21.15.16:0/2283027579 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7ff320065430 0x7ff3200678e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ff32c009f80 tx=0x7ff32c009da0 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:00:25.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:25.753+0000 7ff31ffff640 1 -- 172.21.15.16:0/2283027579 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7ff3280503b0 con 0x7ff3381022f0 2024-06-18T03:00:25.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:25.960+0000 7ff33e160640 1 -- 172.21.15.16:0/2283027579 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff3380fbb00 con 0x7ff3381022f0 2024-06-18T03:00:25.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:25.961+0000 7ff31ffff640 1 -- 172.21.15.16:0/2283027579 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7ff3380fbb00 con 0x7ff3381022f0 2024-06-18T03:00:25.961 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T03:00:25.961 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-18T03:00:25.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:25.962+0000 7ff33e160640 1 -- 172.21.15.16:0/2283027579 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7ff320065430 msgr2=0x7ff3200678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:00:25.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:25.962+0000 7ff33e160640 1 --2- 172.21.15.16:0/2283027579 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7ff320065430 0x7ff3200678e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ff32c009f80 tx=0x7ff32c009da0 comp rx=0 tx=0).stop 2024-06-18T03:00:25.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:25.962+0000 7ff33e160640 1 -- 172.21.15.16:0/2283027579 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3381022f0 msgr2=0x7ff338113000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:00:25.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:25.962+0000 7ff33e160640 1 --2- 172.21.15.16:0/2283027579 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3381022f0 0x7ff338113000 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7ff32800b790 tx=0x7ff3280091d0 comp rx=0 tx=0).stop 2024-06-18T03:00:25.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:25.963+0000 7ff33e160640 1 -- 172.21.15.16:0/2283027579 shutdown_connections 2024-06-18T03:00:25.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:25.963+0000 7ff33e160640 1 --2- 172.21.15.16:0/2283027579 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7ff320065430 0x7ff3200678e0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:00:25.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:25.963+0000 7ff33e160640 1 --2- 172.21.15.16:0/2283027579 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff338102c90 0x7ff338113540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:00:25.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:25.963+0000 7ff33e160640 1 --2- 172.21.15.16:0/2283027579 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3381022f0 0x7ff338113000 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:00:25.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:25.963+0000 7ff33e160640 1 -- 172.21.15.16:0/2283027579 >> 172.21.15.16:0/2283027579 conn(0x7ff3380f7f00 msgr2=0x7ff3380f8e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:00:25.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:25.963+0000 7ff33e160640 1 -- 172.21.15.16:0/2283027579 shutdown_connections 2024-06-18T03:00:25.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:25.963+0000 7ff33e160640 1 -- 172.21.15.16:0/2283027579 wait complete. 2024-06-18T03:00:26.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:00:26 smithi016 ceph-mon[29214]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:00:26.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:00:26 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/2283027579' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:00:26.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:00:26 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T03:00:26.700 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:00:26 smithi123 ceph-mon[36358]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:00:26.700 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:00:26 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/2283027579' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:00:26.700 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:00:26 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T03:00:28.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:00:28 smithi123 ceph-mon[36358]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:00:28.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:00:28 smithi016 ceph-mon[29214]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:00:29.715 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T03:00:29.993 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T03:00:30.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:00:30 smithi123 ceph-mon[36358]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:00:30.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:00:30 smithi016 ceph-mon[29214]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:00:31.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:31.738+0000 7fe2ad109640 1 -- 172.21.15.16:0/3288166544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2a80fb9f0 msgr2=0x7fe2a8108750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:00:31.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:31.738+0000 7fe2ad109640 1 --2- 172.21.15.16:0/3288166544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2a80fb9f0 0x7fe2a8108750 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fe2900048d0 tx=0x7fe290009da0 comp rx=0 tx=0).stop 2024-06-18T03:00:31.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:31.738+0000 7fe2ad109640 1 -- 172.21.15.16:0/3288166544 shutdown_connections 2024-06-18T03:00:31.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:31.738+0000 7fe2ad109640 1 --2- 172.21.15.16:0/3288166544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2a80fb9f0 0x7fe2a8108750 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:00:31.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:31.738+0000 7fe2ad109640 1 --2- 172.21.15.16:0/3288166544 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe2a80fb0e0 0x7fe2a80fb4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:00:31.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:31.738+0000 7fe2ad109640 1 -- 172.21.15.16:0/3288166544 >> 172.21.15.16:0/3288166544 conn(0x7fe2a80f6df0 msgr2=0x7fe2a80f9200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:00:31.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:31.738+0000 7fe2ad109640 1 -- 172.21.15.16:0/3288166544 shutdown_connections 2024-06-18T03:00:31.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:31.738+0000 7fe2ad109640 1 -- 172.21.15.16:0/3288166544 wait complete. 2024-06-18T03:00:31.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:31.739+0000 7fe2ad109640 1 Processor -- start 2024-06-18T03:00:31.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:31.739+0000 7fe2ad109640 1 -- start start 2024-06-18T03:00:31.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:31.739+0000 7fe2ad109640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2a80fb0e0 0x7fe2a8115230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:00:31.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:31.739+0000 7fe2ad109640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe2a80fb9f0 0x7fe2a8115770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:00:31.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:31.739+0000 7fe2ad109640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2a8115d70 con 0x7fe2a80fb0e0 2024-06-18T03:00:31.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:31.739+0000 7fe2ad109640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2a810f200 con 0x7fe2a80fb9f0 2024-06-18T03:00:31.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:31.739+0000 7fe2a6d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2a80fb0e0 0x7fe2a8115230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:00:31.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:31.739+0000 7fe2a6d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2a80fb0e0 0x7fe2a8115230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55038/0 (socket says 172.21.15.16:55038) 2024-06-18T03:00:31.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:31.739+0000 7fe2a6d9d640 1 -- 172.21.15.16:0/428848755 learned_addr learned my addr 172.21.15.16:0/428848755 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T03:00:31.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:31.740+0000 7fe29ffff640 1 --2- 172.21.15.16:0/428848755 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe2a80fb9f0 0x7fe2a8115770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:00:31.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:31.740+0000 7fe2a6d9d640 1 -- 172.21.15.16:0/428848755 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe2a80fb9f0 msgr2=0x7fe2a8115770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:00:31.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:31.740+0000 7fe2a6d9d640 1 --2- 172.21.15.16:0/428848755 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe2a80fb9f0 0x7fe2a8115770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:00:31.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:31.740+0000 7fe2a6d9d640 1 -- 172.21.15.16:0/428848755 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe290005040 con 0x7fe2a80fb0e0 2024-06-18T03:00:31.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:31.740+0000 7fe29ffff640 1 --2- 172.21.15.16:0/428848755 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe2a80fb9f0 0x7fe2a8115770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-18T03:00:31.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:31.740+0000 7fe2a6d9d640 1 --2- 172.21.15.16:0/428848755 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2a80fb0e0 0x7fe2a8115230 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fe29800eb70 tx=0x7fe29800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:00:31.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:31.740+0000 7fe29f7fe640 1 -- 172.21.15.16:0/428848755 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe298007c30 con 0x7fe2a80fb0e0 2024-06-18T03:00:31.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:31.740+0000 7fe29f7fe640 1 -- 172.21.15.16:0/428848755 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe29800f040 con 0x7fe2a80fb0e0 2024-06-18T03:00:31.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:31.740+0000 7fe2ad109640 1 -- 172.21.15.16:0/428848755 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2a810f480 con 0x7fe2a80fb0e0 2024-06-18T03:00:31.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:31.740+0000 7fe29f7fe640 1 -- 172.21.15.16:0/428848755 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe2980095b0 con 0x7fe2a80fb0e0 2024-06-18T03:00:31.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:31.740+0000 7fe2ad109640 1 -- 172.21.15.16:0/428848755 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2a810fa20 con 0x7fe2a80fb0e0 2024-06-18T03:00:31.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:31.741+0000 7fe2ad109640 1 -- 172.21.15.16:0/428848755 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2a8066030 con 0x7fe2a80fb0e0 2024-06-18T03:00:31.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:31.742+0000 7fe29f7fe640 1 -- 172.21.15.16:0/428848755 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fe298009730 con 0x7fe2a80fb0e0 2024-06-18T03:00:31.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:31.742+0000 7fe29f7fe640 1 --2- 172.21.15.16:0/428848755 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fe294065430 0x7fe2940678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:00:31.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:31.742+0000 7fe29f7fe640 1 -- 172.21.15.16:0/428848755 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fe2980821f0 con 0x7fe2a80fb0e0 2024-06-18T03:00:31.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:31.742+0000 7fe29ffff640 1 --2- 172.21.15.16:0/428848755 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fe294065430 0x7fe2940678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:00:31.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:31.743+0000 7fe29ffff640 1 --2- 172.21.15.16:0/428848755 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fe294065430 0x7fe2940678e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe2a8110af0 tx=0x7fe290009da0 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:00:31.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:31.746+0000 7fe29f7fe640 1 -- 172.21.15.16:0/428848755 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fe2980506a0 con 0x7fe2a80fb0e0 2024-06-18T03:00:31.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:31.955+0000 7fe2ad109640 1 -- 172.21.15.16:0/428848755 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe2a810b5b0 con 0x7fe2a80fb0e0 2024-06-18T03:00:31.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:31.956+0000 7fe29f7fe640 1 -- 172.21.15.16:0/428848755 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fe298053cc0 con 0x7fe2a80fb0e0 2024-06-18T03:00:31.956 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T03:00:31.956 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-18T03:00:31.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:31.958+0000 7fe2ad109640 1 -- 172.21.15.16:0/428848755 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fe294065430 msgr2=0x7fe2940678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:00:31.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:31.958+0000 7fe2ad109640 1 --2- 172.21.15.16:0/428848755 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fe294065430 0x7fe2940678e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe2a8110af0 tx=0x7fe290009da0 comp rx=0 tx=0).stop 2024-06-18T03:00:31.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:31.958+0000 7fe2ad109640 1 -- 172.21.15.16:0/428848755 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2a80fb0e0 msgr2=0x7fe2a8115230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:00:31.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:31.958+0000 7fe2ad109640 1 --2- 172.21.15.16:0/428848755 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2a80fb0e0 0x7fe2a8115230 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fe29800eb70 tx=0x7fe29800b040 comp rx=0 tx=0).stop 2024-06-18T03:00:31.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:31.958+0000 7fe2ad109640 1 -- 172.21.15.16:0/428848755 shutdown_connections 2024-06-18T03:00:31.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:31.958+0000 7fe2ad109640 1 --2- 172.21.15.16:0/428848755 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fe294065430 0x7fe2940678e0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:00:31.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:31.958+0000 7fe2ad109640 1 --2- 172.21.15.16:0/428848755 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe2a80fb9f0 0x7fe2a8115770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:00:31.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:31.958+0000 7fe2ad109640 1 --2- 172.21.15.16:0/428848755 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe2a80fb0e0 0x7fe2a8115230 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:00:31.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:31.958+0000 7fe2ad109640 1 -- 172.21.15.16:0/428848755 >> 172.21.15.16:0/428848755 conn(0x7fe2a80f6df0 msgr2=0x7fe2a80f7320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:00:31.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:31.958+0000 7fe2ad109640 1 -- 172.21.15.16:0/428848755 shutdown_connections 2024-06-18T03:00:31.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:31.958+0000 7fe2ad109640 1 -- 172.21.15.16:0/428848755 wait complete. 2024-06-18T03:00:32.676 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:00:32 smithi016 ceph-mon[29214]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:00:32.676 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:00:32 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/428848755' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:00:32.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:00:32 smithi123 ceph-mon[36358]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:00:32.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:00:32 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/428848755' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:00:34.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:00:34 smithi016 ceph-mon[29214]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:00:34.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:00:34 smithi123 ceph-mon[36358]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:00:35.707 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T03:00:35.982 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T03:00:36.699 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:00:36 smithi123 ceph-mon[36358]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:00:36.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:00:36 smithi016 ceph-mon[29214]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:00:37.662 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:00:37 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T03:00:37.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:37.661+0000 7f885e1ae640 1 -- 172.21.15.16:0/3729580515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88580fb570 msgr2=0x7f88580fb940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:00:37.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:37.661+0000 7f885e1ae640 1 --2- 172.21.15.16:0/3729580515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88580fb570 0x7f88580fb940 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f88480048d0 tx=0x7f8848009da0 comp rx=0 tx=0).stop 2024-06-18T03:00:37.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:37.661+0000 7f885e1ae640 1 -- 172.21.15.16:0/3729580515 shutdown_connections 2024-06-18T03:00:37.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:37.661+0000 7f885e1ae640 1 --2- 172.21.15.16:0/3729580515 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f88580fbe80 0x7f88580fcd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:00:37.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:37.661+0000 7f885e1ae640 1 --2- 172.21.15.16:0/3729580515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88580fb570 0x7f88580fb940 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:00:37.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:37.661+0000 7f885e1ae640 1 -- 172.21.15.16:0/3729580515 >> 172.21.15.16:0/3729580515 conn(0x7f8858072110 msgr2=0x7f8858072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:00:37.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:37.661+0000 7f885e1ae640 1 -- 172.21.15.16:0/3729580515 shutdown_connections 2024-06-18T03:00:37.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:37.661+0000 7f885e1ae640 1 -- 172.21.15.16:0/3729580515 wait complete. 2024-06-18T03:00:37.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:37.661+0000 7f885e1ae640 1 Processor -- start 2024-06-18T03:00:37.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:37.662+0000 7f885e1ae640 1 -- start start 2024-06-18T03:00:37.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:37.662+0000 7f885e1ae640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88580fb570 0x7f8858113020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:00:37.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:37.662+0000 7f885e1ae640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f88580fbe80 0x7f8858113560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:00:37.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:37.662+0000 7f885e1ae640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8858113b60 con 0x7f88580fb570 2024-06-18T03:00:37.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:37.662+0000 7f885e1ae640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f885810cfa0 con 0x7f88580fbe80 2024-06-18T03:00:37.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:37.662+0000 7f88577fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88580fb570 0x7f8858113020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:00:37.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:37.662+0000 7f88577fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88580fb570 0x7f8858113020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55068/0 (socket says 172.21.15.16:55068) 2024-06-18T03:00:37.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:37.662+0000 7f88577fe640 1 -- 172.21.15.16:0/810104728 learned_addr learned my addr 172.21.15.16:0/810104728 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T03:00:37.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:37.662+0000 7f884effd640 1 --2- 172.21.15.16:0/810104728 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f88580fbe80 0x7f8858113560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:00:37.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:37.662+0000 7f88577fe640 1 -- 172.21.15.16:0/810104728 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f88580fbe80 msgr2=0x7f8858113560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:00:37.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:37.662+0000 7f88577fe640 1 --2- 172.21.15.16:0/810104728 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f88580fbe80 0x7f8858113560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:00:37.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:37.662+0000 7f88577fe640 1 -- 172.21.15.16:0/810104728 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8848005040 con 0x7f88580fb570 2024-06-18T03:00:37.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:37.662+0000 7f884effd640 1 --2- 172.21.15.16:0/810104728 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f88580fbe80 0x7f8858113560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-18T03:00:37.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:37.662+0000 7f88577fe640 1 --2- 172.21.15.16:0/810104728 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88580fb570 0x7f8858113020 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f8848003080 tx=0x7f8848009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:00:37.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:37.663+0000 7f8854ff9640 1 -- 172.21.15.16:0/810104728 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8848038070 con 0x7f88580fb570 2024-06-18T03:00:37.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:37.663+0000 7f885e1ae640 1 -- 172.21.15.16:0/810104728 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f885810d220 con 0x7f88580fb570 2024-06-18T03:00:37.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:37.663+0000 7f8854ff9640 1 -- 172.21.15.16:0/810104728 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8848032970 con 0x7f88580fb570 2024-06-18T03:00:37.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:37.663+0000 7f8854ff9640 1 -- 172.21.15.16:0/810104728 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f884803d960 con 0x7f88580fb570 2024-06-18T03:00:37.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:37.663+0000 7f885e1ae640 1 -- 172.21.15.16:0/810104728 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f885810d760 con 0x7f88580fb570 2024-06-18T03:00:37.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:37.664+0000 7f885e1ae640 1 -- 172.21.15.16:0/810104728 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88580fc920 con 0x7f88580fb570 2024-06-18T03:00:37.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:37.664+0000 7f8854ff9640 1 -- 172.21.15.16:0/810104728 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f8848030020 con 0x7f88580fb570 2024-06-18T03:00:37.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:37.665+0000 7f8854ff9640 1 --2- 172.21.15.16:0/810104728 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f8844065430 0x7f88440678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:00:37.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:37.665+0000 7f8854ff9640 1 -- 172.21.15.16:0/810104728 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f88480a2630 con 0x7f88580fb570 2024-06-18T03:00:37.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:37.665+0000 7f884effd640 1 --2- 172.21.15.16:0/810104728 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f8844065430 0x7f88440678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:00:37.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:37.665+0000 7f884effd640 1 --2- 172.21.15.16:0/810104728 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f8844065430 0x7f88440678e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f885810e8e0 tx=0x7f884000b040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:00:37.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:37.669+0000 7f8854ff9640 1 -- 172.21.15.16:0/810104728 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f8848070ae0 con 0x7f88580fb570 2024-06-18T03:00:37.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:00:37 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T03:00:37.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:37.878+0000 7f885e1ae640 1 -- 172.21.15.16:0/810104728 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f885804b480 con 0x7f88580fb570 2024-06-18T03:00:37.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:37.878+0000 7f8854ff9640 1 -- 172.21.15.16:0/810104728 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f8848074100 con 0x7f88580fb570 2024-06-18T03:00:37.879 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T03:00:37.879 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-18T03:00:37.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:37.880+0000 7f885e1ae640 1 -- 172.21.15.16:0/810104728 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f8844065430 msgr2=0x7f88440678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:00:37.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:37.880+0000 7f885e1ae640 1 --2- 172.21.15.16:0/810104728 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f8844065430 0x7f88440678e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f885810e8e0 tx=0x7f884000b040 comp rx=0 tx=0).stop 2024-06-18T03:00:37.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:37.880+0000 7f885e1ae640 1 -- 172.21.15.16:0/810104728 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88580fb570 msgr2=0x7f8858113020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:00:37.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:37.880+0000 7f885e1ae640 1 --2- 172.21.15.16:0/810104728 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88580fb570 0x7f8858113020 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f8848003080 tx=0x7f8848009da0 comp rx=0 tx=0).stop 2024-06-18T03:00:37.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:37.880+0000 7f885e1ae640 1 -- 172.21.15.16:0/810104728 shutdown_connections 2024-06-18T03:00:37.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:37.880+0000 7f885e1ae640 1 --2- 172.21.15.16:0/810104728 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f8844065430 0x7f88440678e0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:00:37.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:37.880+0000 7f885e1ae640 1 --2- 172.21.15.16:0/810104728 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f88580fbe80 0x7f8858113560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:00:37.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:37.880+0000 7f885e1ae640 1 --2- 172.21.15.16:0/810104728 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88580fb570 0x7f8858113020 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:00:37.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:37.880+0000 7f885e1ae640 1 -- 172.21.15.16:0/810104728 >> 172.21.15.16:0/810104728 conn(0x7f8858072110 msgr2=0x7f88580ffea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:00:37.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:37.880+0000 7f885e1ae640 1 -- 172.21.15.16:0/810104728 shutdown_connections 2024-06-18T03:00:37.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:37.880+0000 7f885e1ae640 1 -- 172.21.15.16:0/810104728 wait complete. 2024-06-18T03:00:38.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:00:38 smithi016 ceph-mon[29214]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:00:38.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:00:38 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T03:00:38.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:00:38 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T03:00:38.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:00:38 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T03:00:38.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:00:38 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/810104728' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:00:38.635 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:00:38 smithi123 ceph-mon[36358]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:00:38.636 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:00:38 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T03:00:38.636 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:00:38 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T03:00:38.636 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:00:38 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T03:00:38.636 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:00:38 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/810104728' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:00:40.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:00:40 smithi123 ceph-mon[36358]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:00:40.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:00:40 smithi016 ceph-mon[29214]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:00:41.630 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T03:00:41.658 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:00:41 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T03:00:41.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:00:41 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T03:00:41.905 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T03:00:42.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:00:42 smithi123 ceph-mon[36358]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:00:42.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:00:42 smithi016 ceph-mon[29214]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:00:43.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:43.609+0000 7f743f205640 1 -- 172.21.15.16:0/2279836287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74380fbaf0 msgr2=0x7f74380fbf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:00:43.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:43.609+0000 7f743f205640 1 --2- 172.21.15.16:0/2279836287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74380fbaf0 0x7f74380fbf40 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f74200048d0 tx=0x7f7420009da0 comp rx=0 tx=0).stop 2024-06-18T03:00:43.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:43.609+0000 7f743f205640 1 -- 172.21.15.16:0/2279836287 shutdown_connections 2024-06-18T03:00:43.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:43.609+0000 7f743f205640 1 --2- 172.21.15.16:0/2279836287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74380fbaf0 0x7f74380fbf40 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:00:43.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:43.609+0000 7f743f205640 1 --2- 172.21.15.16:0/2279836287 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7438101250 0x7f7438101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:00:43.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:43.609+0000 7f743f205640 1 -- 172.21.15.16:0/2279836287 >> 172.21.15.16:0/2279836287 conn(0x7f7438070a40 msgr2=0x7f7438070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:00:43.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:43.609+0000 7f743f205640 1 -- 172.21.15.16:0/2279836287 shutdown_connections 2024-06-18T03:00:43.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:43.609+0000 7f743f205640 1 -- 172.21.15.16:0/2279836287 wait complete. 2024-06-18T03:00:43.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:43.610+0000 7f743f205640 1 Processor -- start 2024-06-18T03:00:43.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:43.610+0000 7f743f205640 1 -- start start 2024-06-18T03:00:43.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:43.610+0000 7f743f205640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74380fbaf0 0x7f7438115320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:00:43.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:43.610+0000 7f743f205640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7438101250 0x7f7438115860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:00:43.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:43.610+0000 7f743f205640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7438115ef0 con 0x7f74380fbaf0 2024-06-18T03:00:43.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:43.610+0000 7f743f205640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f743810f2a0 con 0x7f7438101250 2024-06-18T03:00:43.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:43.610+0000 7f743cfa1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74380fbaf0 0x7f7438115320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:00:43.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:43.610+0000 7f743cfa1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74380fbaf0 0x7f7438115320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59254/0 (socket says 172.21.15.16:59254) 2024-06-18T03:00:43.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:43.610+0000 7f743cfa1640 1 -- 172.21.15.16:0/1314450548 learned_addr learned my addr 172.21.15.16:0/1314450548 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T03:00:43.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:43.610+0000 7f7437fff640 1 --2- 172.21.15.16:0/1314450548 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7438101250 0x7f7438115860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:00:43.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:43.611+0000 7f743cfa1640 1 -- 172.21.15.16:0/1314450548 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7438101250 msgr2=0x7f7438115860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:00:43.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:43.611+0000 7f743cfa1640 1 --2- 172.21.15.16:0/1314450548 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7438101250 0x7f7438115860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:00:43.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:43.611+0000 7f743cfa1640 1 -- 172.21.15.16:0/1314450548 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7420005040 con 0x7f74380fbaf0 2024-06-18T03:00:43.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:43.611+0000 7f7437fff640 1 --2- 172.21.15.16:0/1314450548 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7438101250 0x7f7438115860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-18T03:00:43.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:43.611+0000 7f743cfa1640 1 --2- 172.21.15.16:0/1314450548 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74380fbaf0 0x7f7438115320 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f742800b790 tx=0x7f74280091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:00:43.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:43.611+0000 7f74357fa640 1 -- 172.21.15.16:0/1314450548 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f742800c700 con 0x7f74380fbaf0 2024-06-18T03:00:43.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:43.611+0000 7f74357fa640 1 -- 172.21.15.16:0/1314450548 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f742800cd30 con 0x7f74380fbaf0 2024-06-18T03:00:43.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:43.611+0000 7f743f205640 1 -- 172.21.15.16:0/1314450548 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f743810f580 con 0x7f74380fbaf0 2024-06-18T03:00:43.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:43.611+0000 7f74357fa640 1 -- 172.21.15.16:0/1314450548 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7428013370 con 0x7f74380fbaf0 2024-06-18T03:00:43.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:43.611+0000 7f743f205640 1 -- 172.21.15.16:0/1314450548 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f743810faa0 con 0x7f74380fbaf0 2024-06-18T03:00:43.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:43.612+0000 7f743f205640 1 -- 172.21.15.16:0/1314450548 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74380fcb00 con 0x7f74380fbaf0 2024-06-18T03:00:43.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:43.613+0000 7f74357fa640 1 -- 172.21.15.16:0/1314450548 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f7428012070 con 0x7f74380fbaf0 2024-06-18T03:00:43.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:43.613+0000 7f74357fa640 1 --2- 172.21.15.16:0/1314450548 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f7424065430 0x7f74240678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:00:43.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:43.613+0000 7f74357fa640 1 -- 172.21.15.16:0/1314450548 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f74280820d0 con 0x7f74380fbaf0 2024-06-18T03:00:43.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:43.613+0000 7f7437fff640 1 --2- 172.21.15.16:0/1314450548 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f7424065430 0x7f74240678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:00:43.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:43.614+0000 7f7437fff640 1 --2- 172.21.15.16:0/1314450548 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f7424065430 0x7f74240678e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7438110be0 tx=0x7f7420035040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:00:43.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:43.617+0000 7f74357fa640 1 -- 172.21.15.16:0/1314450548 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f7428050580 con 0x7f74380fbaf0 2024-06-18T03:00:43.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:43.831+0000 7f743f205640 1 -- 172.21.15.16:0/1314450548 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f743804b4b0 con 0x7f74380fbaf0 2024-06-18T03:00:43.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:43.832+0000 7f74357fa640 1 -- 172.21.15.16:0/1314450548 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f7428053ba0 con 0x7f74380fbaf0 2024-06-18T03:00:43.832 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T03:00:43.832 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-18T03:00:43.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:43.834+0000 7f743f205640 1 -- 172.21.15.16:0/1314450548 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f7424065430 msgr2=0x7f74240678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:00:43.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:43.834+0000 7f743f205640 1 --2- 172.21.15.16:0/1314450548 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f7424065430 0x7f74240678e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7438110be0 tx=0x7f7420035040 comp rx=0 tx=0).stop 2024-06-18T03:00:43.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:43.834+0000 7f743f205640 1 -- 172.21.15.16:0/1314450548 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74380fbaf0 msgr2=0x7f7438115320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:00:43.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:43.834+0000 7f743f205640 1 --2- 172.21.15.16:0/1314450548 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74380fbaf0 0x7f7438115320 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f742800b790 tx=0x7f74280091d0 comp rx=0 tx=0).stop 2024-06-18T03:00:43.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:43.834+0000 7f743f205640 1 -- 172.21.15.16:0/1314450548 shutdown_connections 2024-06-18T03:00:43.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:43.834+0000 7f743f205640 1 --2- 172.21.15.16:0/1314450548 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f7424065430 0x7f74240678e0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:00:43.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:43.834+0000 7f743f205640 1 --2- 172.21.15.16:0/1314450548 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7438101250 0x7f7438115860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:00:43.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:43.834+0000 7f743f205640 1 --2- 172.21.15.16:0/1314450548 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74380fbaf0 0x7f7438115320 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:00:43.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:43.834+0000 7f743f205640 1 -- 172.21.15.16:0/1314450548 >> 172.21.15.16:0/1314450548 conn(0x7f7438070a40 msgr2=0x7f74380f8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:00:43.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:43.834+0000 7f743f205640 1 -- 172.21.15.16:0/1314450548 shutdown_connections 2024-06-18T03:00:43.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:43.834+0000 7f743f205640 1 -- 172.21.15.16:0/1314450548 wait complete. 2024-06-18T03:00:44.484 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:00:44 smithi016 ceph-mon[29214]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:00:44.484 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:00:44 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/1314450548' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:00:44.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:00:44 smithi123 ceph-mon[36358]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:00:44.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:00:44 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/1314450548' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:00:46.700 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:00:46 smithi123 ceph-mon[36358]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:00:46.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:00:46 smithi016 ceph-mon[29214]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:00:47.513 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T03:00:47.791 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T03:00:48.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:00:48 smithi016 ceph-mon[29214]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:00:48.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:00:48 smithi123 ceph-mon[36358]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:00:49.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:49.459+0000 7fd207795640 1 -- 172.21.15.16:0/889644877 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd200102c90 msgr2=0x7fd2000fa9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:00:49.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:49.459+0000 7fd207795640 1 --2- 172.21.15.16:0/889644877 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd200102c90 0x7fd2000fa9e0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fd1f00048d0 tx=0x7fd1f0009da0 comp rx=0 tx=0).stop 2024-06-18T03:00:49.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:49.459+0000 7fd207795640 1 -- 172.21.15.16:0/889644877 shutdown_connections 2024-06-18T03:00:49.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:49.459+0000 7fd207795640 1 --2- 172.21.15.16:0/889644877 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd200102c90 0x7fd2000fa9e0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:00:49.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:49.459+0000 7fd207795640 1 --2- 172.21.15.16:0/889644877 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2001022f0 0x7fd2001026c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:00:49.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:49.459+0000 7fd207795640 1 -- 172.21.15.16:0/889644877 >> 172.21.15.16:0/889644877 conn(0x7fd200070a40 msgr2=0x7fd200070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:00:49.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:49.459+0000 7fd207795640 1 -- 172.21.15.16:0/889644877 shutdown_connections 2024-06-18T03:00:49.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:49.459+0000 7fd207795640 1 -- 172.21.15.16:0/889644877 wait complete. 2024-06-18T03:00:49.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:49.460+0000 7fd207795640 1 Processor -- start 2024-06-18T03:00:49.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:49.460+0000 7fd207795640 1 -- start start 2024-06-18T03:00:49.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:49.460+0000 7fd207795640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2001022f0 0x7fd200115380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:00:49.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:49.460+0000 7fd207795640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd200102c90 0x7fd2001158c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:00:49.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:49.460+0000 7fd207795640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd200115f50 con 0x7fd2001022f0 2024-06-18T03:00:49.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:49.460+0000 7fd207795640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd20010f300 con 0x7fd200102c90 2024-06-18T03:00:49.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:49.460+0000 7fd205531640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2001022f0 0x7fd200115380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:00:49.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:49.460+0000 7fd205531640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2001022f0 0x7fd200115380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59274/0 (socket says 172.21.15.16:59274) 2024-06-18T03:00:49.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:49.460+0000 7fd205531640 1 -- 172.21.15.16:0/712201383 learned_addr learned my addr 172.21.15.16:0/712201383 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T03:00:49.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:49.460+0000 7fd204d30640 1 --2- 172.21.15.16:0/712201383 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd200102c90 0x7fd2001158c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:00:49.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:49.460+0000 7fd205531640 1 -- 172.21.15.16:0/712201383 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd200102c90 msgr2=0x7fd2001158c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:00:49.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:49.460+0000 7fd205531640 1 --2- 172.21.15.16:0/712201383 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd200102c90 0x7fd2001158c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:00:49.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:49.460+0000 7fd205531640 1 -- 172.21.15.16:0/712201383 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1f0005040 con 0x7fd2001022f0 2024-06-18T03:00:49.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:49.461+0000 7fd205531640 1 --2- 172.21.15.16:0/712201383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2001022f0 0x7fd200115380 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc00b580 tx=0x7fd1fc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:00:49.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:49.461+0000 7fd1f5ffb640 1 -- 172.21.15.16:0/712201383 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd1fc00f570 con 0x7fd2001022f0 2024-06-18T03:00:49.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:49.461+0000 7fd207795640 1 -- 172.21.15.16:0/712201383 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd20010f5e0 con 0x7fd2001022f0 2024-06-18T03:00:49.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:49.461+0000 7fd1f5ffb640 1 -- 172.21.15.16:0/712201383 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd1fc00fba0 con 0x7fd2001022f0 2024-06-18T03:00:49.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:49.461+0000 7fd1f5ffb640 1 -- 172.21.15.16:0/712201383 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd1fc010dd0 con 0x7fd2001022f0 2024-06-18T03:00:49.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:49.461+0000 7fd207795640 1 -- 172.21.15.16:0/712201383 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd20010fad0 con 0x7fd2001022f0 2024-06-18T03:00:49.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:49.462+0000 7fd207795640 1 -- 172.21.15.16:0/712201383 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd200065510 con 0x7fd2001022f0 2024-06-18T03:00:49.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:49.462+0000 7fd1f5ffb640 1 -- 172.21.15.16:0/712201383 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fd1fc00f6d0 con 0x7fd2001022f0 2024-06-18T03:00:49.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:49.463+0000 7fd1f5ffb640 1 --2- 172.21.15.16:0/712201383 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fd1ec065430 0x7fd1ec0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:00:49.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:49.463+0000 7fd1f5ffb640 1 -- 172.21.15.16:0/712201383 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fd1fc082400 con 0x7fd2001022f0 2024-06-18T03:00:49.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:49.463+0000 7fd204d30640 1 --2- 172.21.15.16:0/712201383 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fd1ec065430 0x7fd1ec0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:00:49.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:49.463+0000 7fd204d30640 1 --2- 172.21.15.16:0/712201383 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fd1ec065430 0x7fd1ec0678e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd1f00048a0 tx=0x7fd1f0035040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:00:49.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:49.467+0000 7fd1f5ffb640 1 -- 172.21.15.16:0/712201383 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fd1fc050930 con 0x7fd2001022f0 2024-06-18T03:00:49.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:49.675+0000 7fd207795640 1 -- 172.21.15.16:0/712201383 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd20004b510 con 0x7fd2001022f0 2024-06-18T03:00:49.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:49.675+0000 7fd1f5ffb640 1 -- 172.21.15.16:0/712201383 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fd1fc082020 con 0x7fd2001022f0 2024-06-18T03:00:49.676 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T03:00:49.676 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-18T03:00:49.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:49.677+0000 7fd207795640 1 -- 172.21.15.16:0/712201383 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fd1ec065430 msgr2=0x7fd1ec0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:00:49.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:49.677+0000 7fd207795640 1 --2- 172.21.15.16:0/712201383 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fd1ec065430 0x7fd1ec0678e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd1f00048a0 tx=0x7fd1f0035040 comp rx=0 tx=0).stop 2024-06-18T03:00:49.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:49.677+0000 7fd207795640 1 -- 172.21.15.16:0/712201383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2001022f0 msgr2=0x7fd200115380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:00:49.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:49.677+0000 7fd207795640 1 --2- 172.21.15.16:0/712201383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2001022f0 0x7fd200115380 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc00b580 tx=0x7fd1fc0091a0 comp rx=0 tx=0).stop 2024-06-18T03:00:49.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:49.677+0000 7fd207795640 1 -- 172.21.15.16:0/712201383 shutdown_connections 2024-06-18T03:00:49.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:49.677+0000 7fd207795640 1 --2- 172.21.15.16:0/712201383 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fd1ec065430 0x7fd1ec0678e0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:00:49.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:49.677+0000 7fd207795640 1 --2- 172.21.15.16:0/712201383 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd200102c90 0x7fd2001158c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:00:49.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:49.677+0000 7fd207795640 1 --2- 172.21.15.16:0/712201383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2001022f0 0x7fd200115380 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:00:49.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:49.677+0000 7fd207795640 1 -- 172.21.15.16:0/712201383 >> 172.21.15.16:0/712201383 conn(0x7fd200070a40 msgr2=0x7fd2001030a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:00:49.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:49.677+0000 7fd207795640 1 -- 172.21.15.16:0/712201383 shutdown_connections 2024-06-18T03:00:49.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:49.677+0000 7fd207795640 1 -- 172.21.15.16:0/712201383 wait complete. 2024-06-18T03:00:50.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:00:50 smithi123 ceph-mon[36358]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:00:50.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:00:50 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/712201383' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:00:50.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:00:50 smithi016 ceph-mon[29214]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:00:50.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:00:50 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/712201383' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:00:52.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:00:52 smithi123 ceph-mon[36358]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:00:52.778 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:00:52 smithi016 ceph-mon[29214]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:00:53.408 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T03:00:53.676 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T03:00:54.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:00:54 smithi123 ceph-mon[36358]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:00:54.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:00:54 smithi016 ceph-mon[29214]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:00:55.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:55.474+0000 7f037ef03640 1 -- 172.21.15.16:0/4000072760 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0378102c60 msgr2=0x7f03780fc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:00:55.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:55.474+0000 7f037ef03640 1 --2- 172.21.15.16:0/4000072760 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0378102c60 0x7f03780fc2f0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f036c0048d0 tx=0x7f036c009da0 comp rx=0 tx=0).stop 2024-06-18T03:00:55.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:55.474+0000 7f037ef03640 1 -- 172.21.15.16:0/4000072760 shutdown_connections 2024-06-18T03:00:55.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:55.474+0000 7f037ef03640 1 --2- 172.21.15.16:0/4000072760 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0378102c60 0x7f03780fc2f0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:00:55.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:55.474+0000 7f037ef03640 1 --2- 172.21.15.16:0/4000072760 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f03781022c0 0x7f0378102690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:00:55.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:55.474+0000 7f037ef03640 1 -- 172.21.15.16:0/4000072760 >> 172.21.15.16:0/4000072760 conn(0x7f03780f7e50 msgr2=0x7f03780fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:00:55.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:55.474+0000 7f037ef03640 1 -- 172.21.15.16:0/4000072760 shutdown_connections 2024-06-18T03:00:55.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:55.474+0000 7f037ef03640 1 -- 172.21.15.16:0/4000072760 wait complete. 2024-06-18T03:00:55.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:55.475+0000 7f037ef03640 1 Processor -- start 2024-06-18T03:00:55.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:55.475+0000 7f037ef03640 1 -- start start 2024-06-18T03:00:55.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:55.475+0000 7f037ef03640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03781022c0 0x7f037806f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:00:55.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:55.475+0000 7f037ef03640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0378102c60 0x7f037806fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:00:55.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:55.475+0000 7f037ef03640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03780701f0 con 0x7f03781022c0 2024-06-18T03:00:55.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:55.475+0000 7f037ef03640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0378070360 con 0x7f0378102c60 2024-06-18T03:00:55.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:55.475+0000 7f037cc9f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03781022c0 0x7f037806f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:00:55.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:55.475+0000 7f037cc9f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03781022c0 0x7f037806f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44032/0 (socket says 172.21.15.16:44032) 2024-06-18T03:00:55.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:55.475+0000 7f037cc9f640 1 -- 172.21.15.16:0/3438023688 learned_addr learned my addr 172.21.15.16:0/3438023688 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T03:00:55.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:55.475+0000 7f0377fff640 1 --2- 172.21.15.16:0/3438023688 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0378102c60 0x7f037806fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:00:55.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:55.475+0000 7f037cc9f640 1 -- 172.21.15.16:0/3438023688 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0378102c60 msgr2=0x7f037806fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:00:55.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:55.475+0000 7f037cc9f640 1 --2- 172.21.15.16:0/3438023688 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0378102c60 0x7f037806fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:00:55.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:55.475+0000 7f037cc9f640 1 -- 172.21.15.16:0/3438023688 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f036c005040 con 0x7f03781022c0 2024-06-18T03:00:55.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:55.475+0000 7f0377fff640 1 --2- 172.21.15.16:0/3438023688 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0378102c60 0x7f037806fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-18T03:00:55.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:55.476+0000 7f037cc9f640 1 --2- 172.21.15.16:0/3438023688 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03781022c0 0x7f037806f770 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f0368002a50 tx=0x7f0368004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:00:55.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:55.476+0000 7f03757fa640 1 -- 172.21.15.16:0/3438023688 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0368007c30 con 0x7f03781022c0 2024-06-18T03:00:55.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:55.476+0000 7f03757fa640 1 -- 172.21.15.16:0/3438023688 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f036800f040 con 0x7f03781022c0 2024-06-18T03:00:55.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:55.476+0000 7f037ef03640 1 -- 172.21.15.16:0/3438023688 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03780705f0 con 0x7f03781022c0 2024-06-18T03:00:55.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:55.476+0000 7f03757fa640 1 -- 172.21.15.16:0/3438023688 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f036800dbe0 con 0x7f03781022c0 2024-06-18T03:00:55.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:55.476+0000 7f037ef03640 1 -- 172.21.15.16:0/3438023688 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0378119a80 con 0x7f03781022c0 2024-06-18T03:00:55.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:55.477+0000 7f037ef03640 1 -- 172.21.15.16:0/3438023688 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03780fda30 con 0x7f03781022c0 2024-06-18T03:00:55.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:55.477+0000 7f03757fa640 1 -- 172.21.15.16:0/3438023688 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f036800dd40 con 0x7f03781022c0 2024-06-18T03:00:55.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:55.478+0000 7f03757fa640 1 --2- 172.21.15.16:0/3438023688 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f0360065430 0x7f03600678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:00:55.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:55.478+0000 7f03757fa640 1 -- 172.21.15.16:0/3438023688 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f03680823b0 con 0x7f03781022c0 2024-06-18T03:00:55.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:55.478+0000 7f0377fff640 1 --2- 172.21.15.16:0/3438023688 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f0360065430 0x7f03600678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:00:55.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:55.478+0000 7f0377fff640 1 --2- 172.21.15.16:0/3438023688 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f0360065430 0x7f03600678e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f036c000c00 tx=0x7f036c035040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:00:55.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:55.482+0000 7f03757fa640 1 -- 172.21.15.16:0/3438023688 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f03680508e0 con 0x7f03781022c0 2024-06-18T03:00:55.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:55.695+0000 7f037ef03640 1 -- 172.21.15.16:0/3438023688 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f0378070e60 con 0x7f03781022c0 2024-06-18T03:00:55.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:55.696+0000 7f03757fa640 1 -- 172.21.15.16:0/3438023688 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f0368053f00 con 0x7f03781022c0 2024-06-18T03:00:55.696 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T03:00:55.696 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-18T03:00:55.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:55.698+0000 7f037ef03640 1 -- 172.21.15.16:0/3438023688 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f0360065430 msgr2=0x7f03600678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:00:55.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:55.698+0000 7f037ef03640 1 --2- 172.21.15.16:0/3438023688 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f0360065430 0x7f03600678e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f036c000c00 tx=0x7f036c035040 comp rx=0 tx=0).stop 2024-06-18T03:00:55.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:55.698+0000 7f037ef03640 1 -- 172.21.15.16:0/3438023688 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03781022c0 msgr2=0x7f037806f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:00:55.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:55.698+0000 7f037ef03640 1 --2- 172.21.15.16:0/3438023688 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03781022c0 0x7f037806f770 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f0368002a50 tx=0x7f0368004600 comp rx=0 tx=0).stop 2024-06-18T03:00:55.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:55.698+0000 7f037ef03640 1 -- 172.21.15.16:0/3438023688 shutdown_connections 2024-06-18T03:00:55.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:55.698+0000 7f037ef03640 1 --2- 172.21.15.16:0/3438023688 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f0360065430 0x7f03600678e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:00:55.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:55.698+0000 7f037ef03640 1 --2- 172.21.15.16:0/3438023688 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0378102c60 0x7f037806fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:00:55.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:55.698+0000 7f037ef03640 1 --2- 172.21.15.16:0/3438023688 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03781022c0 0x7f037806f770 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:00:55.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:55.698+0000 7f037ef03640 1 -- 172.21.15.16:0/3438023688 >> 172.21.15.16:0/3438023688 conn(0x7f03780f7e50 msgr2=0x7f03780f8db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:00:55.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:55.698+0000 7f037ef03640 1 -- 172.21.15.16:0/3438023688 shutdown_connections 2024-06-18T03:00:55.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:00:55.698+0000 7f037ef03640 1 -- 172.21.15.16:0/3438023688 wait complete. 2024-06-18T03:00:56.699 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:00:56 smithi123 ceph-mon[36358]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:00:56.699 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:00:56 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/3438023688' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:00:56.699 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:00:56 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T03:00:56.733 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:00:56 smithi016 ceph-mon[29214]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:00:56.733 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:00:56 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/3438023688' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:00:56.733 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:00:56 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T03:00:58.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:00:58 smithi123 ceph-mon[36358]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:00:58.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:00:58 smithi016 ceph-mon[29214]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:00:59.466 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T03:00:59.747 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T03:01:00.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:01:00 smithi123 ceph-mon[36358]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:01:00.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:01:00 smithi016 ceph-mon[29214]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:01:01.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:01.404+0000 7fd0c0fbd640 1 -- 172.21.15.16:0/4190353853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0bc0fa9e0 msgr2=0x7fd0bc0fae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:01:01.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:01.404+0000 7fd0c0fbd640 1 --2- 172.21.15.16:0/4190353853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0bc0fa9e0 0x7fd0bc0fae50 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fd0a40048d0 tx=0x7fd0a4009da0 comp rx=0 tx=0).stop 2024-06-18T03:01:01.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:01.404+0000 7fd0c0fbd640 1 -- 172.21.15.16:0/4190353853 shutdown_connections 2024-06-18T03:01:01.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:01.404+0000 7fd0c0fbd640 1 --2- 172.21.15.16:0/4190353853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0bc0fa9e0 0x7fd0bc0fae50 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:01:01.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:01.404+0000 7fd0c0fbd640 1 --2- 172.21.15.16:0/4190353853 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd0bc102370 0x7fd0bc102740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:01:01.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:01.404+0000 7fd0c0fbd640 1 -- 172.21.15.16:0/4190353853 >> 172.21.15.16:0/4190353853 conn(0x7fd0bc070a40 msgr2=0x7fd0bc070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:01:01.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:01.404+0000 7fd0c0fbd640 1 -- 172.21.15.16:0/4190353853 shutdown_connections 2024-06-18T03:01:01.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:01.404+0000 7fd0c0fbd640 1 -- 172.21.15.16:0/4190353853 wait complete. 2024-06-18T03:01:01.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:01.405+0000 7fd0c0fbd640 1 Processor -- start 2024-06-18T03:01:01.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:01.405+0000 7fd0c0fbd640 1 -- start start 2024-06-18T03:01:01.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:01.405+0000 7fd0c0fbd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0bc0fa9e0 0x7fd0bc113010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:01:01.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:01.405+0000 7fd0c0fbd640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd0bc102370 0x7fd0bc113550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:01:01.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:01.405+0000 7fd0c0fbd640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0bc113b50 con 0x7fd0bc0fa9e0 2024-06-18T03:01:01.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:01.405+0000 7fd0c0fbd640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0bc10cf90 con 0x7fd0bc102370 2024-06-18T03:01:01.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:01.405+0000 7fd0ba59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0bc0fa9e0 0x7fd0bc113010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:01:01.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:01.405+0000 7fd0ba59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0bc0fa9e0 0x7fd0bc113010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51026/0 (socket says 172.21.15.16:51026) 2024-06-18T03:01:01.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:01.405+0000 7fd0ba59c640 1 -- 172.21.15.16:0/4022632170 learned_addr learned my addr 172.21.15.16:0/4022632170 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T03:01:01.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:01.406+0000 7fd0ba59c640 1 -- 172.21.15.16:0/4022632170 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd0bc102370 msgr2=0x7fd0bc113550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:01:01.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:01.406+0000 7fd0b3fff640 1 --2- 172.21.15.16:0/4022632170 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd0bc102370 0x7fd0bc113550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:01:01.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:01.406+0000 7fd0ba59c640 1 --2- 172.21.15.16:0/4022632170 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd0bc102370 0x7fd0bc113550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:01:01.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:01.406+0000 7fd0ba59c640 1 -- 172.21.15.16:0/4022632170 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0a4005040 con 0x7fd0bc0fa9e0 2024-06-18T03:01:01.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:01.406+0000 7fd0b3fff640 1 --2- 172.21.15.16:0/4022632170 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd0bc102370 0x7fd0bc113550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-18T03:01:01.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:01.406+0000 7fd0ba59c640 1 --2- 172.21.15.16:0/4022632170 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0bc0fa9e0 0x7fd0bc113010 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fd0ac002a50 tx=0x7fd0ac004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:01:01.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:01.406+0000 7fd0b2ffd640 1 -- 172.21.15.16:0/4022632170 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd0ac007c30 con 0x7fd0bc0fa9e0 2024-06-18T03:01:01.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:01.406+0000 7fd0c0fbd640 1 -- 172.21.15.16:0/4022632170 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0bc10d270 con 0x7fd0bc0fa9e0 2024-06-18T03:01:01.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:01.406+0000 7fd0b2ffd640 1 -- 172.21.15.16:0/4022632170 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd0ac00f040 con 0x7fd0bc0fa9e0 2024-06-18T03:01:01.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:01.406+0000 7fd0b2ffd640 1 -- 172.21.15.16:0/4022632170 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd0ac00dbe0 con 0x7fd0bc0fa9e0 2024-06-18T03:01:01.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:01.406+0000 7fd0c0fbd640 1 -- 172.21.15.16:0/4022632170 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0bc10d790 con 0x7fd0bc0fa9e0 2024-06-18T03:01:01.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:01.407+0000 7fd0c0fbd640 1 -- 172.21.15.16:0/4022632170 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0bc065530 con 0x7fd0bc0fa9e0 2024-06-18T03:01:01.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:01.408+0000 7fd0b2ffd640 1 -- 172.21.15.16:0/4022632170 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fd0ac00dd40 con 0x7fd0bc0fa9e0 2024-06-18T03:01:01.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:01.408+0000 7fd0b2ffd640 1 --2- 172.21.15.16:0/4022632170 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fd0a8065430 0x7fd0a80678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:01:01.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:01.408+0000 7fd0b2ffd640 1 -- 172.21.15.16:0/4022632170 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fd0ac082470 con 0x7fd0bc0fa9e0 2024-06-18T03:01:01.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:01.408+0000 7fd0b3fff640 1 --2- 172.21.15.16:0/4022632170 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fd0a8065430 0x7fd0a80678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:01:01.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:01.409+0000 7fd0b3fff640 1 --2- 172.21.15.16:0/4022632170 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fd0a8065430 0x7fd0a80678e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd0a4009f80 tx=0x7fd0a4009da0 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:01:01.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:01.412+0000 7fd0b2ffd640 1 -- 172.21.15.16:0/4022632170 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fd0ac050920 con 0x7fd0bc0fa9e0 2024-06-18T03:01:01.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:01.625+0000 7fd0c0fbd640 1 -- 172.21.15.16:0/4022632170 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd0bc074960 con 0x7fd0bc0fa9e0 2024-06-18T03:01:01.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:01.626+0000 7fd0b2ffd640 1 -- 172.21.15.16:0/4022632170 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fd0bc074960 con 0x7fd0bc0fa9e0 2024-06-18T03:01:01.626 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T03:01:01.627 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-18T03:01:01.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:01.628+0000 7fd0c0fbd640 1 -- 172.21.15.16:0/4022632170 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fd0a8065430 msgr2=0x7fd0a80678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:01:01.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:01.628+0000 7fd0c0fbd640 1 --2- 172.21.15.16:0/4022632170 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fd0a8065430 0x7fd0a80678e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd0a4009f80 tx=0x7fd0a4009da0 comp rx=0 tx=0).stop 2024-06-18T03:01:01.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:01.628+0000 7fd0c0fbd640 1 -- 172.21.15.16:0/4022632170 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0bc0fa9e0 msgr2=0x7fd0bc113010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:01:01.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:01.628+0000 7fd0c0fbd640 1 --2- 172.21.15.16:0/4022632170 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0bc0fa9e0 0x7fd0bc113010 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fd0ac002a50 tx=0x7fd0ac004600 comp rx=0 tx=0).stop 2024-06-18T03:01:01.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:01.628+0000 7fd0c0fbd640 1 -- 172.21.15.16:0/4022632170 shutdown_connections 2024-06-18T03:01:01.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:01.628+0000 7fd0c0fbd640 1 --2- 172.21.15.16:0/4022632170 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fd0a8065430 0x7fd0a80678e0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:01:01.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:01.628+0000 7fd0c0fbd640 1 --2- 172.21.15.16:0/4022632170 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd0bc102370 0x7fd0bc113550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:01:01.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:01.628+0000 7fd0c0fbd640 1 --2- 172.21.15.16:0/4022632170 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0bc0fa9e0 0x7fd0bc113010 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:01:01.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:01.628+0000 7fd0c0fbd640 1 -- 172.21.15.16:0/4022632170 >> 172.21.15.16:0/4022632170 conn(0x7fd0bc070a40 msgr2=0x7fd0bc0fe3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:01:01.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:01.628+0000 7fd0c0fbd640 1 -- 172.21.15.16:0/4022632170 shutdown_connections 2024-06-18T03:01:01.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:01.628+0000 7fd0c0fbd640 1 -- 172.21.15.16:0/4022632170 wait complete. 2024-06-18T03:01:02.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:01:02 smithi016 ceph-mon[29214]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:01:02.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:01:02 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/4022632170' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:01:02.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:01:02 smithi123 ceph-mon[36358]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:01:02.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:01:02 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/4022632170' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:01:04.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:01:04 smithi016 ceph-mon[29214]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:01:04.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:01:04 smithi123 ceph-mon[36358]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:01:05.318 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T03:01:05.599 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T03:01:06.699 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:01:06 smithi123 ceph-mon[36358]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:01:06.718 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:01:06 smithi016 ceph-mon[29214]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:01:07.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:07.197+0000 7f321f33e640 1 -- 172.21.15.16:0/1440270866 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3218101200 msgr2=0x7f32181015d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:01:07.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:07.197+0000 7f321f33e640 1 --2- 172.21.15.16:0/1440270866 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3218101200 0x7f32181015d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f32140048d0 tx=0x7f3214009da0 comp rx=0 tx=0).stop 2024-06-18T03:01:07.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:07.197+0000 7f321f33e640 1 -- 172.21.15.16:0/1440270866 shutdown_connections 2024-06-18T03:01:07.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:07.197+0000 7f321f33e640 1 --2- 172.21.15.16:0/1440270866 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3218101ba0 0x7f32180fb230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:01:07.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:07.197+0000 7f321f33e640 1 --2- 172.21.15.16:0/1440270866 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3218101200 0x7f32181015d0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:01:07.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:07.197+0000 7f321f33e640 1 -- 172.21.15.16:0/1440270866 >> 172.21.15.16:0/1440270866 conn(0x7f32180f6df0 msgr2=0x7f32180f9200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:01:07.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:07.198+0000 7f321f33e640 1 -- 172.21.15.16:0/1440270866 shutdown_connections 2024-06-18T03:01:07.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:07.198+0000 7f321f33e640 1 -- 172.21.15.16:0/1440270866 wait complete. 2024-06-18T03:01:07.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:07.198+0000 7f321f33e640 1 Processor -- start 2024-06-18T03:01:07.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:07.198+0000 7f321f33e640 1 -- start start 2024-06-18T03:01:07.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:07.198+0000 7f321f33e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3218101200 0x7f3218102080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:01:07.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:07.198+0000 7f321f33e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3218101ba0 0x7f32181025c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:01:07.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:07.198+0000 7f321f33e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3218102b00 con 0x7f3218101ba0 2024-06-18T03:01:07.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:07.198+0000 7f321f33e640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3218102c70 con 0x7f3218101200 2024-06-18T03:01:07.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:07.198+0000 7f321c8d9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3218101ba0 0x7f32181025c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:01:07.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:07.199+0000 7f321c8d9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3218101ba0 0x7f32181025c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51038/0 (socket says 172.21.15.16:51038) 2024-06-18T03:01:07.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:07.199+0000 7f321c8d9640 1 -- 172.21.15.16:0/372749518 learned_addr learned my addr 172.21.15.16:0/372749518 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T03:01:07.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:07.199+0000 7f321d0da640 1 --2- 172.21.15.16:0/372749518 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3218101200 0x7f3218102080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:01:07.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:07.199+0000 7f321c8d9640 1 -- 172.21.15.16:0/372749518 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3218101200 msgr2=0x7f3218102080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:01:07.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:07.199+0000 7f321c8d9640 1 --2- 172.21.15.16:0/372749518 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3218101200 0x7f3218102080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:01:07.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:07.199+0000 7f321c8d9640 1 -- 172.21.15.16:0/372749518 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3214005040 con 0x7f3218101ba0 2024-06-18T03:01:07.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:07.199+0000 7f321c8d9640 1 --2- 172.21.15.16:0/372749518 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3218101ba0 0x7f32181025c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f320800c790 tx=0x7f32080043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:01:07.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:07.199+0000 7f320dffb640 1 -- 172.21.15.16:0/372749518 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f320800a7a0 con 0x7f3218101ba0 2024-06-18T03:01:07.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:07.199+0000 7f320dffb640 1 -- 172.21.15.16:0/372749518 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f320800add0 con 0x7f3218101ba0 2024-06-18T03:01:07.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:07.199+0000 7f321f33e640 1 -- 172.21.15.16:0/372749518 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3218102f00 con 0x7f3218101ba0 2024-06-18T03:01:07.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:07.199+0000 7f320dffb640 1 -- 172.21.15.16:0/372749518 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f32080115e0 con 0x7f3218101ba0 2024-06-18T03:01:07.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:07.199+0000 7f321f33e640 1 -- 172.21.15.16:0/372749518 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3218119740 con 0x7f3218101ba0 2024-06-18T03:01:07.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:07.200+0000 7f321f33e640 1 -- 172.21.15.16:0/372749518 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32180fc970 con 0x7f3218101ba0 2024-06-18T03:01:07.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:07.201+0000 7f320dffb640 1 -- 172.21.15.16:0/372749518 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f320800a900 con 0x7f3218101ba0 2024-06-18T03:01:07.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:07.201+0000 7f320dffb640 1 --2- 172.21.15.16:0/372749518 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f3204065430 0x7f32040678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:01:07.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:07.201+0000 7f320dffb640 1 -- 172.21.15.16:0/372749518 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f3208082440 con 0x7f3218101ba0 2024-06-18T03:01:07.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:07.201+0000 7f321d0da640 1 --2- 172.21.15.16:0/372749518 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f3204065430 0x7f32040678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:01:07.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:07.202+0000 7f321d0da640 1 --2- 172.21.15.16:0/372749518 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f3204065430 0x7f32040678e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f3214009f80 tx=0x7f3214035040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:01:07.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:07.206+0000 7f320dffb640 1 -- 172.21.15.16:0/372749518 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f3208050a10 con 0x7f3218101ba0 2024-06-18T03:01:07.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:07.423+0000 7f321f33e640 1 -- 172.21.15.16:0/372749518 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f3218103770 con 0x7f3218101ba0 2024-06-18T03:01:07.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:07.423+0000 7f320dffb640 1 -- 172.21.15.16:0/372749518 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f3208054030 con 0x7f3218101ba0 2024-06-18T03:01:07.424 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T03:01:07.424 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-18T03:01:07.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:07.425+0000 7f321f33e640 1 -- 172.21.15.16:0/372749518 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f3204065430 msgr2=0x7f32040678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:01:07.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:07.425+0000 7f321f33e640 1 --2- 172.21.15.16:0/372749518 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f3204065430 0x7f32040678e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f3214009f80 tx=0x7f3214035040 comp rx=0 tx=0).stop 2024-06-18T03:01:07.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:07.425+0000 7f321f33e640 1 -- 172.21.15.16:0/372749518 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3218101ba0 msgr2=0x7f32181025c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:01:07.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:07.425+0000 7f321f33e640 1 --2- 172.21.15.16:0/372749518 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3218101ba0 0x7f32181025c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f320800c790 tx=0x7f32080043c0 comp rx=0 tx=0).stop 2024-06-18T03:01:07.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:07.425+0000 7f321f33e640 1 -- 172.21.15.16:0/372749518 shutdown_connections 2024-06-18T03:01:07.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:07.425+0000 7f321f33e640 1 --2- 172.21.15.16:0/372749518 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f3204065430 0x7f32040678e0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:01:07.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:07.425+0000 7f321f33e640 1 --2- 172.21.15.16:0/372749518 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3218101ba0 0x7f32181025c0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:01:07.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:07.425+0000 7f321f33e640 1 --2- 172.21.15.16:0/372749518 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3218101200 0x7f3218102080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:01:07.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:07.425+0000 7f321f33e640 1 -- 172.21.15.16:0/372749518 >> 172.21.15.16:0/372749518 conn(0x7f32180f6df0 msgr2=0x7f32180f8480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:01:07.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:07.425+0000 7f321f33e640 1 -- 172.21.15.16:0/372749518 shutdown_connections 2024-06-18T03:01:07.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:07.425+0000 7f321f33e640 1 -- 172.21.15.16:0/372749518 wait complete. 2024-06-18T03:01:08.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:01:08 smithi016 ceph-mon[29214]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:01:08.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:01:08 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/372749518' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:01:08.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:01:08 smithi123 ceph-mon[36358]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:01:08.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:01:08 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/372749518' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:01:10.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:01:10 smithi123 ceph-mon[36358]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:01:10.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:01:10 smithi016 ceph-mon[29214]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:01:11.211 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T03:01:11.497 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T03:01:11.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:01:11 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T03:01:11.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:01:11 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T03:01:12.699 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:01:12 smithi016 ceph-mon[29214]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:01:12.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:01:12 smithi123 ceph-mon[36358]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:01:13.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:13.201+0000 7f3faba15640 1 -- 172.21.15.16:0/2229800833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3fa40fbe80 msgr2=0x7f3fa40fcd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:01:13.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:13.201+0000 7f3faba15640 1 --2- 172.21.15.16:0/2229800833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3fa40fbe80 0x7f3fa40fcd80 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f3f940048d0 tx=0x7f3f94009da0 comp rx=0 tx=0).stop 2024-06-18T03:01:13.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:13.201+0000 7f3faba15640 1 -- 172.21.15.16:0/2229800833 shutdown_connections 2024-06-18T03:01:13.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:13.201+0000 7f3faba15640 1 --2- 172.21.15.16:0/2229800833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3fa40fbe80 0x7f3fa40fcd80 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:01:13.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:13.201+0000 7f3faba15640 1 --2- 172.21.15.16:0/2229800833 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3fa40fb570 0x7f3fa40fb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:01:13.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:13.201+0000 7f3faba15640 1 -- 172.21.15.16:0/2229800833 >> 172.21.15.16:0/2229800833 conn(0x7f3fa4072110 msgr2=0x7f3fa4072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:01:13.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:13.201+0000 7f3faba15640 1 -- 172.21.15.16:0/2229800833 shutdown_connections 2024-06-18T03:01:13.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:13.201+0000 7f3faba15640 1 -- 172.21.15.16:0/2229800833 wait complete. 2024-06-18T03:01:13.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:13.202+0000 7f3faba15640 1 Processor -- start 2024-06-18T03:01:13.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:13.202+0000 7f3faba15640 1 -- start start 2024-06-18T03:01:13.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:13.202+0000 7f3faba15640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3fa40fb570 0x7f3fa4111ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:01:13.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:13.202+0000 7f3faba15640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3fa40fbe80 0x7f3fa4112410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:01:13.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:13.202+0000 7f3faba15640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fa4112a10 con 0x7f3fa40fbe80 2024-06-18T03:01:13.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:13.202+0000 7f3faba15640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fa410be50 con 0x7f3fa40fb570 2024-06-18T03:01:13.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:13.202+0000 7f3fa8fb0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3fa40fbe80 0x7f3fa4112410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:01:13.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:13.202+0000 7f3fa8fb0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3fa40fbe80 0x7f3fa4112410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60698/0 (socket says 172.21.15.16:60698) 2024-06-18T03:01:13.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:13.202+0000 7f3fa8fb0640 1 -- 172.21.15.16:0/2778124990 learned_addr learned my addr 172.21.15.16:0/2778124990 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T03:01:13.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:13.202+0000 7f3fa97b1640 1 --2- 172.21.15.16:0/2778124990 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3fa40fb570 0x7f3fa4111ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:01:13.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:13.202+0000 7f3fa8fb0640 1 -- 172.21.15.16:0/2778124990 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3fa40fb570 msgr2=0x7f3fa4111ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:01:13.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:13.202+0000 7f3fa8fb0640 1 --2- 172.21.15.16:0/2778124990 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3fa40fb570 0x7f3fa4111ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:01:13.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:13.202+0000 7f3fa8fb0640 1 -- 172.21.15.16:0/2778124990 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f94005040 con 0x7f3fa40fbe80 2024-06-18T03:01:13.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:13.203+0000 7f3fa8fb0640 1 --2- 172.21.15.16:0/2778124990 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3fa40fbe80 0x7f3fa4112410 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f3f94003080 tx=0x7f3f94009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:01:13.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:13.203+0000 7f3f99ffb640 1 -- 172.21.15.16:0/2778124990 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f94038070 con 0x7f3fa40fbe80 2024-06-18T03:01:13.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:13.203+0000 7f3faba15640 1 -- 172.21.15.16:0/2778124990 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fa410c0d0 con 0x7f3fa40fbe80 2024-06-18T03:01:13.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:13.203+0000 7f3f99ffb640 1 -- 172.21.15.16:0/2778124990 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3f94008ee0 con 0x7f3fa40fbe80 2024-06-18T03:01:13.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:13.203+0000 7f3f99ffb640 1 -- 172.21.15.16:0/2778124990 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f9403d870 con 0x7f3fa40fbe80 2024-06-18T03:01:13.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:13.203+0000 7f3faba15640 1 -- 172.21.15.16:0/2778124990 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fa410c610 con 0x7f3fa40fbe80 2024-06-18T03:01:13.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:13.204+0000 7f3faba15640 1 -- 172.21.15.16:0/2778124990 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fa40fc920 con 0x7f3fa40fbe80 2024-06-18T03:01:13.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:13.205+0000 7f3f99ffb640 1 -- 172.21.15.16:0/2778124990 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f3f94030020 con 0x7f3fa40fbe80 2024-06-18T03:01:13.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:13.205+0000 7f3f99ffb640 1 --2- 172.21.15.16:0/2778124990 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f3f90065430 0x7f3f900678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:01:13.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:13.205+0000 7f3f99ffb640 1 -- 172.21.15.16:0/2778124990 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f3f940a2250 con 0x7f3fa40fbe80 2024-06-18T03:01:13.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:13.205+0000 7f3fa97b1640 1 --2- 172.21.15.16:0/2778124990 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f3f90065430 0x7f3f900678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:01:13.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:13.206+0000 7f3fa97b1640 1 --2- 172.21.15.16:0/2778124990 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f3f90065430 0x7f3f900678e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f3fa0004550 tx=0x7f3fa0009040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:01:13.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:13.209+0000 7f3f99ffb640 1 -- 172.21.15.16:0/2778124990 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f3f94070780 con 0x7f3fa40fbe80 2024-06-18T03:01:13.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:13.419+0000 7f3faba15640 1 -- 172.21.15.16:0/2778124990 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f3fa410d540 con 0x7f3fa40fbe80 2024-06-18T03:01:13.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:13.420+0000 7f3f99ffb640 1 -- 172.21.15.16:0/2778124990 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f3f94073da0 con 0x7f3fa40fbe80 2024-06-18T03:01:13.420 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T03:01:13.420 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-18T03:01:13.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:13.422+0000 7f3faba15640 1 -- 172.21.15.16:0/2778124990 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f3f90065430 msgr2=0x7f3f900678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:01:13.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:13.422+0000 7f3faba15640 1 --2- 172.21.15.16:0/2778124990 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f3f90065430 0x7f3f900678e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f3fa0004550 tx=0x7f3fa0009040 comp rx=0 tx=0).stop 2024-06-18T03:01:13.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:13.422+0000 7f3faba15640 1 -- 172.21.15.16:0/2778124990 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3fa40fbe80 msgr2=0x7f3fa4112410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:01:13.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:13.422+0000 7f3faba15640 1 --2- 172.21.15.16:0/2778124990 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3fa40fbe80 0x7f3fa4112410 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f3f94003080 tx=0x7f3f94009da0 comp rx=0 tx=0).stop 2024-06-18T03:01:13.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:13.422+0000 7f3faba15640 1 -- 172.21.15.16:0/2778124990 shutdown_connections 2024-06-18T03:01:13.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:13.422+0000 7f3faba15640 1 --2- 172.21.15.16:0/2778124990 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f3f90065430 0x7f3f900678e0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:01:13.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:13.422+0000 7f3faba15640 1 --2- 172.21.15.16:0/2778124990 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3fa40fbe80 0x7f3fa4112410 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:01:13.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:13.422+0000 7f3faba15640 1 --2- 172.21.15.16:0/2778124990 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3fa40fb570 0x7f3fa4111ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:01:13.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:13.422+0000 7f3faba15640 1 -- 172.21.15.16:0/2778124990 >> 172.21.15.16:0/2778124990 conn(0x7f3fa4072110 msgr2=0x7f3fa40ffea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:01:13.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:13.422+0000 7f3faba15640 1 -- 172.21.15.16:0/2778124990 shutdown_connections 2024-06-18T03:01:13.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:13.422+0000 7f3faba15640 1 -- 172.21.15.16:0/2778124990 wait complete. 2024-06-18T03:01:13.707 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:01:13 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/2778124990' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:01:13.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:01:13 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/2778124990' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:01:14.688 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:01:14 smithi016 ceph-mon[29214]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:01:14.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:01:14 smithi123 ceph-mon[36358]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:01:16.699 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:01:16 smithi123 ceph-mon[36358]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:01:16.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:01:16 smithi016 ceph-mon[29214]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:01:17.158 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T03:01:17.437 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T03:01:18.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:01:18 smithi123 ceph-mon[36358]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:01:18.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:01:18 smithi016 ceph-mon[29214]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:01:19.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:19.146+0000 7f325285a640 1 -- 172.21.15.16:0/127752138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f324c0fa760 msgr2=0x7f324c0fab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:01:19.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:19.146+0000 7f325285a640 1 --2- 172.21.15.16:0/127752138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f324c0fa760 0x7f324c0fab30 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f323c004900 tx=0x7f323c009da0 comp rx=0 tx=0).stop 2024-06-18T03:01:19.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:19.147+0000 7f325285a640 1 -- 172.21.15.16:0/127752138 shutdown_connections 2024-06-18T03:01:19.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:19.147+0000 7f325285a640 1 --2- 172.21.15.16:0/127752138 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f324c0fb100 0x7f324c106140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:01:19.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:19.147+0000 7f325285a640 1 --2- 172.21.15.16:0/127752138 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f324c0fa760 0x7f324c0fab30 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:01:19.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:19.147+0000 7f325285a640 1 -- 172.21.15.16:0/127752138 >> 172.21.15.16:0/127752138 conn(0x7f324c0724a0 msgr2=0x7f324c0728a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:01:19.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:19.147+0000 7f325285a640 1 -- 172.21.15.16:0/127752138 shutdown_connections 2024-06-18T03:01:19.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:19.147+0000 7f325285a640 1 -- 172.21.15.16:0/127752138 wait complete. 2024-06-18T03:01:19.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:19.147+0000 7f325285a640 1 Processor -- start 2024-06-18T03:01:19.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:19.147+0000 7f325285a640 1 -- start start 2024-06-18T03:01:19.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:19.148+0000 7f325285a640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f324c0fa760 0x7f324c113170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:01:19.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:19.148+0000 7f325285a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f324c0fb100 0x7f324c1136b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:01:19.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:19.148+0000 7f325285a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f324c113d40 con 0x7f324c0fb100 2024-06-18T03:01:19.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:19.148+0000 7f325285a640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f324c10d0f0 con 0x7f324c0fa760 2024-06-18T03:01:19.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:19.148+0000 7f324b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f324c0fb100 0x7f324c1136b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:01:19.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:19.148+0000 7f324b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f324c0fb100 0x7f324c1136b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60706/0 (socket says 172.21.15.16:60706) 2024-06-18T03:01:19.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:19.148+0000 7f324b7fe640 1 -- 172.21.15.16:0/3471552791 learned_addr learned my addr 172.21.15.16:0/3471552791 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T03:01:19.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:19.148+0000 7f324bfff640 1 --2- 172.21.15.16:0/3471552791 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f324c0fa760 0x7f324c113170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:01:19.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:19.148+0000 7f324b7fe640 1 -- 172.21.15.16:0/3471552791 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f324c0fa760 msgr2=0x7f324c113170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:01:19.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:19.148+0000 7f324b7fe640 1 --2- 172.21.15.16:0/3471552791 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f324c0fa760 0x7f324c113170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:01:19.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:19.148+0000 7f324b7fe640 1 -- 172.21.15.16:0/3471552791 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f323c005040 con 0x7f324c0fb100 2024-06-18T03:01:19.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:19.148+0000 7f324b7fe640 1 --2- 172.21.15.16:0/3471552791 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f324c0fb100 0x7f324c1136b0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f324000b580 tx=0x7f32400091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:01:19.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:19.149+0000 7f3248ff9640 1 -- 172.21.15.16:0/3471552791 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f324000f570 con 0x7f324c0fb100 2024-06-18T03:01:19.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:19.149+0000 7f3248ff9640 1 -- 172.21.15.16:0/3471552791 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f324000fba0 con 0x7f324c0fb100 2024-06-18T03:01:19.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:19.149+0000 7f325285a640 1 -- 172.21.15.16:0/3471552791 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f324c10d3d0 con 0x7f324c0fb100 2024-06-18T03:01:19.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:19.149+0000 7f3248ff9640 1 -- 172.21.15.16:0/3471552791 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3240010dd0 con 0x7f324c0fb100 2024-06-18T03:01:19.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:19.149+0000 7f325285a640 1 -- 172.21.15.16:0/3471552791 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f324c10d8f0 con 0x7f324c0fb100 2024-06-18T03:01:19.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:19.151+0000 7f3248ff9640 1 -- 172.21.15.16:0/3471552791 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f324000f6d0 con 0x7f324c0fb100 2024-06-18T03:01:19.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:19.151+0000 7f325285a640 1 -- 172.21.15.16:0/3471552791 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f324c0fb810 con 0x7f324c0fb100 2024-06-18T03:01:19.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:19.151+0000 7f3248ff9640 1 --2- 172.21.15.16:0/3471552791 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f3234065430 0x7f32340678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:01:19.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:19.152+0000 7f3248ff9640 1 -- 172.21.15.16:0/3471552791 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f3240082460 con 0x7f324c0fb100 2024-06-18T03:01:19.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:19.152+0000 7f324bfff640 1 --2- 172.21.15.16:0/3471552791 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f3234065430 0x7f32340678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:01:19.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:19.152+0000 7f324bfff640 1 --2- 172.21.15.16:0/3471552791 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f3234065430 0x7f32340678e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f323c009f80 tx=0x7f323c009da0 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:01:19.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:19.156+0000 7f3248ff9640 1 -- 172.21.15.16:0/3471552791 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f3240050930 con 0x7f324c0fb100 2024-06-18T03:01:19.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:19.368+0000 7f325285a640 1 -- 172.21.15.16:0/3471552791 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f324c072fb0 con 0x7f324c0fb100 2024-06-18T03:01:19.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:19.368+0000 7f3248ff9640 1 -- 172.21.15.16:0/3471552791 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f3240082070 con 0x7f324c0fb100 2024-06-18T03:01:19.369 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T03:01:19.369 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-18T03:01:19.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:19.371+0000 7f325285a640 1 -- 172.21.15.16:0/3471552791 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f3234065430 msgr2=0x7f32340678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:01:19.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:19.371+0000 7f325285a640 1 --2- 172.21.15.16:0/3471552791 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f3234065430 0x7f32340678e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f323c009f80 tx=0x7f323c009da0 comp rx=0 tx=0).stop 2024-06-18T03:01:19.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:19.371+0000 7f325285a640 1 -- 172.21.15.16:0/3471552791 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f324c0fb100 msgr2=0x7f324c1136b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:01:19.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:19.371+0000 7f325285a640 1 --2- 172.21.15.16:0/3471552791 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f324c0fb100 0x7f324c1136b0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f324000b580 tx=0x7f32400091a0 comp rx=0 tx=0).stop 2024-06-18T03:01:19.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:19.371+0000 7f325285a640 1 -- 172.21.15.16:0/3471552791 shutdown_connections 2024-06-18T03:01:19.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:19.371+0000 7f325285a640 1 --2- 172.21.15.16:0/3471552791 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f3234065430 0x7f32340678e0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:01:19.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:19.371+0000 7f325285a640 1 --2- 172.21.15.16:0/3471552791 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f324c0fb100 0x7f324c1136b0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:01:19.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:19.371+0000 7f325285a640 1 --2- 172.21.15.16:0/3471552791 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f324c0fa760 0x7f324c113170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:01:19.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:19.371+0000 7f325285a640 1 -- 172.21.15.16:0/3471552791 >> 172.21.15.16:0/3471552791 conn(0x7f324c0724a0 msgr2=0x7f324c0f8ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:01:19.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:19.371+0000 7f325285a640 1 -- 172.21.15.16:0/3471552791 shutdown_connections 2024-06-18T03:01:19.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:19.371+0000 7f325285a640 1 -- 172.21.15.16:0/3471552791 wait complete. 2024-06-18T03:01:19.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:01:19 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/3471552791' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:01:19.737 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:01:19 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/3471552791' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:01:20.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:01:20 smithi123 ceph-mon[36358]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:01:20.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:01:20 smithi016 ceph-mon[29214]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:01:22.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:01:22 smithi123 ceph-mon[36358]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:01:22.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:01:22 smithi016 ceph-mon[29214]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:01:23.025 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T03:01:23.301 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T03:01:24.729 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:01:24 smithi016 ceph-mon[29214]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:01:24.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:01:24 smithi123 ceph-mon[36358]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:01:25.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:25.094+0000 7ff05c7c0640 1 -- 172.21.15.16:0/1160317263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff054102c40 msgr2=0x7ff0540fc2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:01:25.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:25.094+0000 7ff05c7c0640 1 --2- 172.21.15.16:0/1160317263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff054102c40 0x7ff0540fc2d0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7ff0500048d0 tx=0x7ff050009da0 comp rx=0 tx=0).stop 2024-06-18T03:01:25.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:25.094+0000 7ff05c7c0640 1 -- 172.21.15.16:0/1160317263 shutdown_connections 2024-06-18T03:01:25.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:25.094+0000 7ff05c7c0640 1 --2- 172.21.15.16:0/1160317263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff054102c40 0x7ff0540fc2d0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:01:25.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:25.094+0000 7ff05c7c0640 1 --2- 172.21.15.16:0/1160317263 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff0541022a0 0x7ff054102670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:01:25.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:25.094+0000 7ff05c7c0640 1 -- 172.21.15.16:0/1160317263 >> 172.21.15.16:0/1160317263 conn(0x7ff0540f7e50 msgr2=0x7ff0540fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:01:25.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:25.094+0000 7ff05c7c0640 1 -- 172.21.15.16:0/1160317263 shutdown_connections 2024-06-18T03:01:25.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:25.094+0000 7ff05c7c0640 1 -- 172.21.15.16:0/1160317263 wait complete. 2024-06-18T03:01:25.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:25.095+0000 7ff05c7c0640 1 Processor -- start 2024-06-18T03:01:25.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:25.095+0000 7ff05c7c0640 1 -- start start 2024-06-18T03:01:25.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:25.095+0000 7ff05c7c0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0541022a0 0x7ff05406f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:01:25.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:25.095+0000 7ff05c7c0640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff054102c40 0x7ff05406fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:01:25.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:25.095+0000 7ff05c7c0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0540701f0 con 0x7ff0541022a0 2024-06-18T03:01:25.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:25.095+0000 7ff05c7c0640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff054070360 con 0x7ff054102c40 2024-06-18T03:01:25.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:25.095+0000 7ff05a55c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0541022a0 0x7ff05406f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:01:25.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:25.095+0000 7ff05a55c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0541022a0 0x7ff05406f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34322/0 (socket says 172.21.15.16:34322) 2024-06-18T03:01:25.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:25.095+0000 7ff05a55c640 1 -- 172.21.15.16:0/1817551708 learned_addr learned my addr 172.21.15.16:0/1817551708 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T03:01:25.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:25.095+0000 7ff05a55c640 1 -- 172.21.15.16:0/1817551708 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff054102c40 msgr2=0x7ff05406fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:01:25.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:25.095+0000 7ff05a55c640 1 --2- 172.21.15.16:0/1817551708 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff054102c40 0x7ff05406fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:01:25.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:25.095+0000 7ff05a55c640 1 -- 172.21.15.16:0/1817551708 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff050005040 con 0x7ff0541022a0 2024-06-18T03:01:25.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:25.095+0000 7ff05a55c640 1 --2- 172.21.15.16:0/1817551708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0541022a0 0x7ff05406f770 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7ff044002a50 tx=0x7ff044004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:01:25.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:25.096+0000 7ff04affd640 1 -- 172.21.15.16:0/1817551708 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff044007c30 con 0x7ff0541022a0 2024-06-18T03:01:25.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:25.096+0000 7ff05c7c0640 1 -- 172.21.15.16:0/1817551708 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0540705f0 con 0x7ff0541022a0 2024-06-18T03:01:25.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:25.096+0000 7ff04affd640 1 -- 172.21.15.16:0/1817551708 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff04400f040 con 0x7ff0541022a0 2024-06-18T03:01:25.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:25.096+0000 7ff04affd640 1 -- 172.21.15.16:0/1817551708 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff04400dbe0 con 0x7ff0541022a0 2024-06-18T03:01:25.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:25.096+0000 7ff05c7c0640 1 -- 172.21.15.16:0/1817551708 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff054119720 con 0x7ff0541022a0 2024-06-18T03:01:25.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:25.097+0000 7ff05c7c0640 1 -- 172.21.15.16:0/1817551708 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0540fda10 con 0x7ff0541022a0 2024-06-18T03:01:25.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:25.097+0000 7ff04affd640 1 -- 172.21.15.16:0/1817551708 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7ff04400dd40 con 0x7ff0541022a0 2024-06-18T03:01:25.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:25.098+0000 7ff04affd640 1 --2- 172.21.15.16:0/1817551708 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7ff040065430 0x7ff0400678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:01:25.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:25.098+0000 7ff04affd640 1 -- 172.21.15.16:0/1817551708 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7ff044082470 con 0x7ff0541022a0 2024-06-18T03:01:25.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:25.098+0000 7ff059d5b640 1 --2- 172.21.15.16:0/1817551708 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7ff040065430 0x7ff0400678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:01:25.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:25.098+0000 7ff059d5b640 1 --2- 172.21.15.16:0/1817551708 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7ff040065430 0x7ff0400678e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ff050032430 tx=0x7ff050009da0 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:01:25.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:25.102+0000 7ff04affd640 1 -- 172.21.15.16:0/1817551708 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7ff044050920 con 0x7ff0541022a0 2024-06-18T03:01:25.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:25.309+0000 7ff05c7c0640 1 -- 172.21.15.16:0/1817551708 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff0540fba90 con 0x7ff0541022a0 2024-06-18T03:01:25.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:25.310+0000 7ff04affd640 1 -- 172.21.15.16:0/1817551708 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7ff044053f40 con 0x7ff0541022a0 2024-06-18T03:01:25.310 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T03:01:25.310 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-18T03:01:25.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:25.311+0000 7ff05c7c0640 1 -- 172.21.15.16:0/1817551708 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7ff040065430 msgr2=0x7ff0400678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:01:25.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:25.311+0000 7ff05c7c0640 1 --2- 172.21.15.16:0/1817551708 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7ff040065430 0x7ff0400678e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ff050032430 tx=0x7ff050009da0 comp rx=0 tx=0).stop 2024-06-18T03:01:25.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:25.311+0000 7ff05c7c0640 1 -- 172.21.15.16:0/1817551708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0541022a0 msgr2=0x7ff05406f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:01:25.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:25.311+0000 7ff05c7c0640 1 --2- 172.21.15.16:0/1817551708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0541022a0 0x7ff05406f770 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7ff044002a50 tx=0x7ff044004600 comp rx=0 tx=0).stop 2024-06-18T03:01:25.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:25.311+0000 7ff05c7c0640 1 -- 172.21.15.16:0/1817551708 shutdown_connections 2024-06-18T03:01:25.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:25.311+0000 7ff05c7c0640 1 --2- 172.21.15.16:0/1817551708 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7ff040065430 0x7ff0400678e0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:01:25.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:25.311+0000 7ff05c7c0640 1 --2- 172.21.15.16:0/1817551708 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff054102c40 0x7ff05406fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:01:25.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:25.311+0000 7ff05c7c0640 1 --2- 172.21.15.16:0/1817551708 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0541022a0 0x7ff05406f770 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:01:25.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:25.311+0000 7ff05c7c0640 1 -- 172.21.15.16:0/1817551708 >> 172.21.15.16:0/1817551708 conn(0x7ff0540f7e50 msgr2=0x7ff0540f8d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:01:25.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:25.312+0000 7ff05c7c0640 1 -- 172.21.15.16:0/1817551708 shutdown_connections 2024-06-18T03:01:25.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:25.312+0000 7ff05c7c0640 1 -- 172.21.15.16:0/1817551708 wait complete. 2024-06-18T03:01:25.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:01:25 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/1817551708' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:01:25.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:01:25 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/1817551708' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:01:26.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:01:26 smithi016 ceph-mon[29214]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:01:26.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:01:26 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T03:01:26.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:01:26 smithi123 ceph-mon[36358]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:01:26.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:01:26 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T03:01:28.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:01:28 smithi123 ceph-mon[36358]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:01:28.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:01:28 smithi016 ceph-mon[29214]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:01:28.946 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T03:01:29.228 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T03:01:30.641 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:01:30 smithi016 ceph-mon[29214]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:01:30.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:01:30 smithi123 ceph-mon[36358]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:01:30.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:30.863+0000 7f1ee31ef640 1 -- 172.21.15.16:0/3186573366 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1edc1011b0 msgr2=0x7f1edc101580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:01:30.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:30.863+0000 7f1ee31ef640 1 --2- 172.21.15.16:0/3186573366 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1edc1011b0 0x7f1edc101580 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f1ed00048d0 tx=0x7f1ed0009da0 comp rx=0 tx=0).stop 2024-06-18T03:01:30.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:30.863+0000 7f1ee31ef640 1 -- 172.21.15.16:0/3186573366 shutdown_connections 2024-06-18T03:01:30.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:30.863+0000 7f1ee31ef640 1 --2- 172.21.15.16:0/3186573366 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1edc101b50 0x7f1edc0fb1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:01:30.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:30.863+0000 7f1ee31ef640 1 --2- 172.21.15.16:0/3186573366 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1edc1011b0 0x7f1edc101580 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:01:30.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:30.863+0000 7f1ee31ef640 1 -- 172.21.15.16:0/3186573366 >> 172.21.15.16:0/3186573366 conn(0x7f1edc0f6d60 msgr2=0x7f1edc0f9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:01:30.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:30.863+0000 7f1ee31ef640 1 -- 172.21.15.16:0/3186573366 shutdown_connections 2024-06-18T03:01:30.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:30.863+0000 7f1ee31ef640 1 -- 172.21.15.16:0/3186573366 wait complete. 2024-06-18T03:01:30.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:30.863+0000 7f1ee31ef640 1 Processor -- start 2024-06-18T03:01:30.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:30.863+0000 7f1ee31ef640 1 -- start start 2024-06-18T03:01:30.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:30.864+0000 7f1ee31ef640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1edc1011b0 0x7f1edc102030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:01:30.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:30.864+0000 7f1ee31ef640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1edc101b50 0x7f1edc102570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:01:30.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:30.864+0000 7f1ee31ef640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1edc102ab0 con 0x7f1edc101b50 2024-06-18T03:01:30.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:30.864+0000 7f1ee31ef640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1edc102c20 con 0x7f1edc1011b0 2024-06-18T03:01:30.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:30.864+0000 7f1edbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1edc101b50 0x7f1edc102570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:01:30.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:30.864+0000 7f1edbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1edc101b50 0x7f1edc102570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51464/0 (socket says 172.21.15.16:51464) 2024-06-18T03:01:30.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:30.864+0000 7f1edbfff640 1 -- 172.21.15.16:0/721698457 learned_addr learned my addr 172.21.15.16:0/721698457 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T03:01:30.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:30.864+0000 7f1ee0f8b640 1 --2- 172.21.15.16:0/721698457 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1edc1011b0 0x7f1edc102030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:01:30.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:30.864+0000 7f1edbfff640 1 -- 172.21.15.16:0/721698457 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1edc1011b0 msgr2=0x7f1edc102030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:01:30.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:30.864+0000 7f1edbfff640 1 --2- 172.21.15.16:0/721698457 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1edc1011b0 0x7f1edc102030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:01:30.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:30.864+0000 7f1edbfff640 1 -- 172.21.15.16:0/721698457 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ed0005040 con 0x7f1edc101b50 2024-06-18T03:01:30.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:30.864+0000 7f1edbfff640 1 --2- 172.21.15.16:0/721698457 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1edc101b50 0x7f1edc102570 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f1ec8002a50 tx=0x7f1ec8004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:01:30.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:30.865+0000 7f1ed97fa640 1 -- 172.21.15.16:0/721698457 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ec8007c30 con 0x7f1edc101b50 2024-06-18T03:01:30.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:30.865+0000 7f1ed97fa640 1 -- 172.21.15.16:0/721698457 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1ec800f040 con 0x7f1edc101b50 2024-06-18T03:01:30.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:30.865+0000 7f1ee31ef640 1 -- 172.21.15.16:0/721698457 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1edc102eb0 con 0x7f1edc101b50 2024-06-18T03:01:30.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:30.865+0000 7f1ed97fa640 1 -- 172.21.15.16:0/721698457 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ec800dad0 con 0x7f1edc101b50 2024-06-18T03:01:30.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:30.865+0000 7f1ee31ef640 1 -- 172.21.15.16:0/721698457 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1edc119770 con 0x7f1edc101b50 2024-06-18T03:01:30.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:30.866+0000 7f1ee31ef640 1 -- 172.21.15.16:0/721698457 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1edc0fc920 con 0x7f1edc101b50 2024-06-18T03:01:30.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:30.866+0000 7f1ed97fa640 1 -- 172.21.15.16:0/721698457 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f1ec800dc70 con 0x7f1edc101b50 2024-06-18T03:01:30.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:30.867+0000 7f1ed97fa640 1 --2- 172.21.15.16:0/721698457 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f1ecc065430 0x7f1ecc0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:01:30.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:30.867+0000 7f1ed97fa640 1 -- 172.21.15.16:0/721698457 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f1ec8081c70 con 0x7f1edc101b50 2024-06-18T03:01:30.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:30.867+0000 7f1ee0f8b640 1 --2- 172.21.15.16:0/721698457 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f1ecc065430 0x7f1ecc0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:01:30.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:30.867+0000 7f1ee0f8b640 1 --2- 172.21.15.16:0/721698457 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f1ecc065430 0x7f1ecc0678e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f1ed0009f80 tx=0x7f1ed0035040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:01:30.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:30.871+0000 7f1ed97fa640 1 -- 172.21.15.16:0/721698457 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f1ec804cdf0 con 0x7f1edc101b50 2024-06-18T03:01:31.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:31.080+0000 7f1ee31ef640 1 -- 172.21.15.16:0/721698457 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f1edc119360 con 0x7f1edc101b50 2024-06-18T03:01:31.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:31.081+0000 7f1ed97fa640 1 -- 172.21.15.16:0/721698457 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f1edc119360 con 0x7f1edc101b50 2024-06-18T03:01:31.082 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T03:01:31.082 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-18T03:01:31.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:31.083+0000 7f1ee31ef640 1 -- 172.21.15.16:0/721698457 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f1ecc065430 msgr2=0x7f1ecc0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:01:31.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:31.083+0000 7f1ee31ef640 1 --2- 172.21.15.16:0/721698457 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f1ecc065430 0x7f1ecc0678e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f1ed0009f80 tx=0x7f1ed0035040 comp rx=0 tx=0).stop 2024-06-18T03:01:31.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:31.083+0000 7f1ee31ef640 1 -- 172.21.15.16:0/721698457 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1edc101b50 msgr2=0x7f1edc102570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:01:31.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:31.083+0000 7f1ee31ef640 1 --2- 172.21.15.16:0/721698457 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1edc101b50 0x7f1edc102570 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f1ec8002a50 tx=0x7f1ec8004600 comp rx=0 tx=0).stop 2024-06-18T03:01:31.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:31.083+0000 7f1ee31ef640 1 -- 172.21.15.16:0/721698457 shutdown_connections 2024-06-18T03:01:31.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:31.083+0000 7f1ee31ef640 1 --2- 172.21.15.16:0/721698457 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f1ecc065430 0x7f1ecc0678e0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:01:31.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:31.083+0000 7f1ee31ef640 1 --2- 172.21.15.16:0/721698457 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1edc101b50 0x7f1edc102570 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:01:31.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:31.083+0000 7f1ee31ef640 1 --2- 172.21.15.16:0/721698457 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1edc1011b0 0x7f1edc102030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:01:31.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:31.083+0000 7f1ee31ef640 1 -- 172.21.15.16:0/721698457 >> 172.21.15.16:0/721698457 conn(0x7f1edc0f6d60 msgr2=0x7f1edc0f7ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:01:31.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:31.083+0000 7f1ee31ef640 1 -- 172.21.15.16:0/721698457 shutdown_connections 2024-06-18T03:01:31.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:31.083+0000 7f1ee31ef640 1 -- 172.21.15.16:0/721698457 wait complete. 2024-06-18T03:01:31.702 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:01:31 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/721698457' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:01:31.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:01:31 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/721698457' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:01:32.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:01:32 smithi123 ceph-mon[36358]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:01:32.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:01:32 smithi016 ceph-mon[29214]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:01:34.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:01:34 smithi016 ceph-mon[29214]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:01:34.730 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T03:01:34.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:01:34 smithi123 ceph-mon[36358]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:01:35.007 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T03:01:36.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:36.608+0000 7f8753e3d640 1 -- 172.21.15.16:0/3144097032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f874c06fd50 msgr2=0x7f874c0701c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:01:36.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:36.608+0000 7f8753e3d640 1 --2- 172.21.15.16:0/3144097032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f874c06fd50 0x7f874c0701c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f873c0048d0 tx=0x7f873c009da0 comp rx=0 tx=0).stop 2024-06-18T03:01:36.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:36.608+0000 7f8753e3d640 1 -- 172.21.15.16:0/3144097032 shutdown_connections 2024-06-18T03:01:36.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:36.608+0000 7f8753e3d640 1 --2- 172.21.15.16:0/3144097032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f874c06fd50 0x7f874c0701c0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:01:36.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:36.608+0000 7f8753e3d640 1 --2- 172.21.15.16:0/3144097032 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f874c071440 0x7f874c06f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:01:36.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:36.608+0000 7f8753e3d640 1 -- 172.21.15.16:0/3144097032 >> 172.21.15.16:0/3144097032 conn(0x7f874c0f7e50 msgr2=0x7f874c0fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:01:36.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:36.608+0000 7f8753e3d640 1 -- 172.21.15.16:0/3144097032 shutdown_connections 2024-06-18T03:01:36.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:36.608+0000 7f8753e3d640 1 -- 172.21.15.16:0/3144097032 wait complete. 2024-06-18T03:01:36.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:36.609+0000 7f8753e3d640 1 Processor -- start 2024-06-18T03:01:36.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:36.609+0000 7f8753e3d640 1 -- start start 2024-06-18T03:01:36.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:36.609+0000 7f8753e3d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f874c06fd50 0x7f874c119980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:01:36.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:36.609+0000 7f8753e3d640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f874c071440 0x7f874c119ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:01:36.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:36.609+0000 7f8753e3d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f874c11a510 con 0x7f874c06fd50 2024-06-18T03:01:36.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:36.609+0000 7f8753e3d640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f874c113920 con 0x7f874c071440 2024-06-18T03:01:36.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:36.609+0000 7f8751bd9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f874c06fd50 0x7f874c119980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:01:36.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:36.609+0000 7f8751bd9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f874c06fd50 0x7f874c119980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51478/0 (socket says 172.21.15.16:51478) 2024-06-18T03:01:36.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:36.609+0000 7f8751bd9640 1 -- 172.21.15.16:0/2220500232 learned_addr learned my addr 172.21.15.16:0/2220500232 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T03:01:36.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:36.609+0000 7f87513d8640 1 --2- 172.21.15.16:0/2220500232 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f874c071440 0x7f874c119ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:01:36.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:36.609+0000 7f8751bd9640 1 -- 172.21.15.16:0/2220500232 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f874c071440 msgr2=0x7f874c119ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:01:36.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:36.609+0000 7f8751bd9640 1 --2- 172.21.15.16:0/2220500232 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f874c071440 0x7f874c119ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:01:36.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:36.609+0000 7f8751bd9640 1 -- 172.21.15.16:0/2220500232 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f873c005040 con 0x7f874c06fd50 2024-06-18T03:01:36.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:36.610+0000 7f8751bd9640 1 --2- 172.21.15.16:0/2220500232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f874c06fd50 0x7f874c119980 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f8748002a50 tx=0x7f8748004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:01:36.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:36.610+0000 7f87427fc640 1 -- 172.21.15.16:0/2220500232 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8748007c30 con 0x7f874c06fd50 2024-06-18T03:01:36.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:36.610+0000 7f87427fc640 1 -- 172.21.15.16:0/2220500232 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f874800f040 con 0x7f874c06fd50 2024-06-18T03:01:36.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:36.610+0000 7f8753e3d640 1 -- 172.21.15.16:0/2220500232 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f874c113c00 con 0x7f874c06fd50 2024-06-18T03:01:36.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:36.610+0000 7f87427fc640 1 -- 172.21.15.16:0/2220500232 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f874800db60 con 0x7f874c06fd50 2024-06-18T03:01:36.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:36.610+0000 7f8753e3d640 1 -- 172.21.15.16:0/2220500232 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f874c1140f0 con 0x7f874c06fd50 2024-06-18T03:01:36.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:36.611+0000 7f8753e3d640 1 -- 172.21.15.16:0/2220500232 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f874c070f50 con 0x7f874c06fd50 2024-06-18T03:01:36.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:36.612+0000 7f87427fc640 1 -- 172.21.15.16:0/2220500232 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f874800dcc0 con 0x7f874c06fd50 2024-06-18T03:01:36.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:36.612+0000 7f87427fc640 1 --2- 172.21.15.16:0/2220500232 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f8738065430 0x7f87380678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:01:36.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:36.612+0000 7f87427fc640 1 -- 172.21.15.16:0/2220500232 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f8748082310 con 0x7f874c06fd50 2024-06-18T03:01:36.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:36.612+0000 7f87513d8640 1 --2- 172.21.15.16:0/2220500232 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f8738065430 0x7f87380678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:01:36.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:36.613+0000 7f87513d8640 1 --2- 172.21.15.16:0/2220500232 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f8738065430 0x7f87380678e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f873c0043d0 tx=0x7f873c035040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:01:36.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:36.616+0000 7f87427fc640 1 -- 172.21.15.16:0/2220500232 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f87480507c0 con 0x7f874c06fd50 2024-06-18T03:01:36.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:01:36 smithi123 ceph-mon[36358]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:01:36.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:01:36 smithi016 ceph-mon[29214]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:01:36.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:36.827+0000 7f8753e3d640 1 -- 172.21.15.16:0/2220500232 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f874c114ff0 con 0x7f874c06fd50 2024-06-18T03:01:36.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:36.827+0000 7f87427fc640 1 -- 172.21.15.16:0/2220500232 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f8748053de0 con 0x7f874c06fd50 2024-06-18T03:01:36.827 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T03:01:36.828 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-18T03:01:36.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:36.829+0000 7f8753e3d640 1 -- 172.21.15.16:0/2220500232 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f8738065430 msgr2=0x7f87380678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:01:36.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:36.829+0000 7f8753e3d640 1 --2- 172.21.15.16:0/2220500232 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f8738065430 0x7f87380678e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f873c0043d0 tx=0x7f873c035040 comp rx=0 tx=0).stop 2024-06-18T03:01:36.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:36.829+0000 7f8753e3d640 1 -- 172.21.15.16:0/2220500232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f874c06fd50 msgr2=0x7f874c119980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:01:36.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:36.829+0000 7f8753e3d640 1 --2- 172.21.15.16:0/2220500232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f874c06fd50 0x7f874c119980 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f8748002a50 tx=0x7f8748004600 comp rx=0 tx=0).stop 2024-06-18T03:01:36.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:36.829+0000 7f8753e3d640 1 -- 172.21.15.16:0/2220500232 shutdown_connections 2024-06-18T03:01:36.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:36.829+0000 7f8753e3d640 1 --2- 172.21.15.16:0/2220500232 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f8738065430 0x7f87380678e0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:01:36.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:36.829+0000 7f8753e3d640 1 --2- 172.21.15.16:0/2220500232 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f874c071440 0x7f874c119ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:01:36.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:36.829+0000 7f8753e3d640 1 --2- 172.21.15.16:0/2220500232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f874c06fd50 0x7f874c119980 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:01:36.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:36.829+0000 7f8753e3d640 1 -- 172.21.15.16:0/2220500232 >> 172.21.15.16:0/2220500232 conn(0x7f874c0f7e50 msgr2=0x7f874c0f9020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:01:36.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:36.829+0000 7f8753e3d640 1 -- 172.21.15.16:0/2220500232 shutdown_connections 2024-06-18T03:01:36.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:36.829+0000 7f8753e3d640 1 -- 172.21.15.16:0/2220500232 wait complete. 2024-06-18T03:01:37.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:01:37 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/2220500232' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:01:37.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:01:37 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/2220500232' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:01:38.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:01:38 smithi123 ceph-mon[36358]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:01:38.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:01:38 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T03:01:38.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:01:38 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T03:01:38.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:01:38 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T03:01:38.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:01:38 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T03:01:38.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:01:38 smithi016 ceph-mon[29214]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:01:38.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:01:38 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T03:01:38.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:01:38 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T03:01:38.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:01:38 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T03:01:38.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:01:38 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T03:01:40.498 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T03:01:40.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:01:40 smithi123 ceph-mon[36358]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:01:40.776 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T03:01:40.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:01:40 smithi016 ceph-mon[29214]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:01:41.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:01:41 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T03:01:41.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:01:41 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T03:01:42.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:42.331+0000 7fed280a8640 1 -- 172.21.15.16:0/675096461 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed201022c0 msgr2=0x7fed20102690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:01:42.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:42.331+0000 7fed280a8640 1 --2- 172.21.15.16:0/675096461 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed201022c0 0x7fed20102690 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fed1c009c50 tx=0x7fed1c004d10 comp rx=0 tx=0).stop 2024-06-18T03:01:42.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:42.331+0000 7fed280a8640 1 -- 172.21.15.16:0/675096461 shutdown_connections 2024-06-18T03:01:42.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:42.331+0000 7fed280a8640 1 --2- 172.21.15.16:0/675096461 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed20102c60 0x7fed200fc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:01:42.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:42.331+0000 7fed280a8640 1 --2- 172.21.15.16:0/675096461 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed201022c0 0x7fed20102690 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:01:42.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:42.331+0000 7fed280a8640 1 -- 172.21.15.16:0/675096461 >> 172.21.15.16:0/675096461 conn(0x7fed200f7e50 msgr2=0x7fed200fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:01:42.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:42.331+0000 7fed280a8640 1 -- 172.21.15.16:0/675096461 shutdown_connections 2024-06-18T03:01:42.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:42.331+0000 7fed280a8640 1 -- 172.21.15.16:0/675096461 wait complete. 2024-06-18T03:01:42.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:42.332+0000 7fed280a8640 1 Processor -- start 2024-06-18T03:01:42.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:42.332+0000 7fed280a8640 1 -- start start 2024-06-18T03:01:42.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:42.332+0000 7fed280a8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed201022c0 0x7fed20115560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:01:42.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:42.332+0000 7fed280a8640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed20102c60 0x7fed20115aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:01:42.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:42.332+0000 7fed280a8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed201160f0 con 0x7fed201022c0 2024-06-18T03:01:42.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:42.332+0000 7fed280a8640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed2010f500 con 0x7fed20102c60 2024-06-18T03:01:42.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:42.332+0000 7fed25e44640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed201022c0 0x7fed20115560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:01:42.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:42.332+0000 7fed25e44640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed201022c0 0x7fed20115560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45020/0 (socket says 172.21.15.16:45020) 2024-06-18T03:01:42.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:42.332+0000 7fed25e44640 1 -- 172.21.15.16:0/2370311914 learned_addr learned my addr 172.21.15.16:0/2370311914 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T03:01:42.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:42.332+0000 7fed25e44640 1 -- 172.21.15.16:0/2370311914 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed20102c60 msgr2=0x7fed20115aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:01:42.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:42.332+0000 7fed25643640 1 --2- 172.21.15.16:0/2370311914 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed20102c60 0x7fed20115aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:01:42.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:42.332+0000 7fed25e44640 1 --2- 172.21.15.16:0/2370311914 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed20102c60 0x7fed20115aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:01:42.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:42.332+0000 7fed25e44640 1 -- 172.21.15.16:0/2370311914 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed1c0098b0 con 0x7fed201022c0 2024-06-18T03:01:42.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:42.332+0000 7fed25e44640 1 --2- 172.21.15.16:0/2370311914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed201022c0 0x7fed20115560 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fed1c004f80 tx=0x7fed1c004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:01:42.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:42.333+0000 7fed167fc640 1 -- 172.21.15.16:0/2370311914 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fed1c008de0 con 0x7fed201022c0 2024-06-18T03:01:42.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:42.333+0000 7fed167fc640 1 -- 172.21.15.16:0/2370311914 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fed1c032c00 con 0x7fed201022c0 2024-06-18T03:01:42.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:42.333+0000 7fed280a8640 1 -- 172.21.15.16:0/2370311914 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed2010f780 con 0x7fed201022c0 2024-06-18T03:01:42.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:42.333+0000 7fed167fc640 1 -- 172.21.15.16:0/2370311914 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fed1c03dbc0 con 0x7fed201022c0 2024-06-18T03:01:42.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:42.333+0000 7fed280a8640 1 -- 172.21.15.16:0/2370311914 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed2010fca0 con 0x7fed201022c0 2024-06-18T03:01:42.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:42.334+0000 7fed280a8640 1 -- 172.21.15.16:0/2370311914 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed200fda30 con 0x7fed201022c0 2024-06-18T03:01:42.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:42.334+0000 7fed167fc640 1 -- 172.21.15.16:0/2370311914 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fed1c02f030 con 0x7fed201022c0 2024-06-18T03:01:42.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:42.335+0000 7fed167fc640 1 --2- 172.21.15.16:0/2370311914 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fed0c065430 0x7fed0c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:01:42.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:42.335+0000 7fed167fc640 1 -- 172.21.15.16:0/2370311914 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fed1c0a3ce0 con 0x7fed201022c0 2024-06-18T03:01:42.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:42.335+0000 7fed25643640 1 --2- 172.21.15.16:0/2370311914 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fed0c065430 0x7fed0c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:01:42.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:42.335+0000 7fed25643640 1 --2- 172.21.15.16:0/2370311914 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fed0c065430 0x7fed0c0678e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fed10007fd0 tx=0x7fed10009040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:01:42.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:42.339+0000 7fed167fc640 1 -- 172.21.15.16:0/2370311914 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fed1c072240 con 0x7fed201022c0 2024-06-18T03:01:42.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:42.550+0000 7fed280a8640 1 -- 172.21.15.16:0/2370311914 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fed20109430 con 0x7fed201022c0 2024-06-18T03:01:42.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:42.550+0000 7fed167fc640 1 -- 172.21.15.16:0/2370311914 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fed1c032d70 con 0x7fed201022c0 2024-06-18T03:01:42.551 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T03:01:42.551 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-18T03:01:42.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:42.552+0000 7fed280a8640 1 -- 172.21.15.16:0/2370311914 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fed0c065430 msgr2=0x7fed0c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:01:42.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:42.552+0000 7fed280a8640 1 --2- 172.21.15.16:0/2370311914 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fed0c065430 0x7fed0c0678e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fed10007fd0 tx=0x7fed10009040 comp rx=0 tx=0).stop 2024-06-18T03:01:42.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:42.552+0000 7fed280a8640 1 -- 172.21.15.16:0/2370311914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed201022c0 msgr2=0x7fed20115560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:01:42.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:42.552+0000 7fed280a8640 1 --2- 172.21.15.16:0/2370311914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed201022c0 0x7fed20115560 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fed1c004f80 tx=0x7fed1c004d10 comp rx=0 tx=0).stop 2024-06-18T03:01:42.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:42.552+0000 7fed280a8640 1 -- 172.21.15.16:0/2370311914 shutdown_connections 2024-06-18T03:01:42.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:42.552+0000 7fed280a8640 1 --2- 172.21.15.16:0/2370311914 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fed0c065430 0x7fed0c0678e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:01:42.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:42.552+0000 7fed280a8640 1 --2- 172.21.15.16:0/2370311914 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed20102c60 0x7fed20115aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:01:42.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:42.552+0000 7fed280a8640 1 --2- 172.21.15.16:0/2370311914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed201022c0 0x7fed20115560 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:01:42.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:42.552+0000 7fed280a8640 1 -- 172.21.15.16:0/2370311914 >> 172.21.15.16:0/2370311914 conn(0x7fed200f7e50 msgr2=0x7fed2006f8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:01:42.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:42.552+0000 7fed280a8640 1 -- 172.21.15.16:0/2370311914 shutdown_connections 2024-06-18T03:01:42.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:42.552+0000 7fed280a8640 1 -- 172.21.15.16:0/2370311914 wait complete. 2024-06-18T03:01:42.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:01:42 smithi016 ceph-mon[29214]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:01:42.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:01:42 smithi123 ceph-mon[36358]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:01:43.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:01:43 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/2370311914' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:01:43.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:01:43 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/2370311914' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:01:44.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:01:44 smithi016 ceph-mon[29214]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:01:44.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:01:44 smithi123 ceph-mon[36358]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:01:46.165 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T03:01:46.443 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T03:01:46.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:01:46 smithi016 ceph-mon[29214]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:01:46.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:01:46 smithi123 ceph-mon[36358]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:01:48.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:48.018+0000 7f48230c8640 1 -- 172.21.15.16:0/3913076544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f481c1022f0 msgr2=0x7f481c1026c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:01:48.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:48.018+0000 7f48230c8640 1 --2- 172.21.15.16:0/3913076544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f481c1022f0 0x7f481c1026c0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f480c0048d0 tx=0x7f480c009da0 comp rx=0 tx=0).stop 2024-06-18T03:01:48.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:48.018+0000 7f48230c8640 1 -- 172.21.15.16:0/3913076544 shutdown_connections 2024-06-18T03:01:48.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:48.018+0000 7f48230c8640 1 --2- 172.21.15.16:0/3913076544 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f481c102c90 0x7f481c0fa9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:01:48.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:48.018+0000 7f48230c8640 1 --2- 172.21.15.16:0/3913076544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f481c1022f0 0x7f481c1026c0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:01:48.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:48.018+0000 7f48230c8640 1 -- 172.21.15.16:0/3913076544 >> 172.21.15.16:0/3913076544 conn(0x7f481c070a40 msgr2=0x7f481c070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:01:48.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:48.018+0000 7f48230c8640 1 -- 172.21.15.16:0/3913076544 shutdown_connections 2024-06-18T03:01:48.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:48.018+0000 7f48230c8640 1 -- 172.21.15.16:0/3913076544 wait complete. 2024-06-18T03:01:48.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:48.018+0000 7f48230c8640 1 Processor -- start 2024-06-18T03:01:48.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:48.018+0000 7f48230c8640 1 -- start start 2024-06-18T03:01:48.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:48.019+0000 7f48230c8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f481c1022f0 0x7f481c0fbbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:01:48.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:48.019+0000 7f48230c8640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f481c102c90 0x7f481c0fc100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:01:48.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:48.019+0000 7f48230c8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f481c0fc640 con 0x7f481c1022f0 2024-06-18T03:01:48.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:48.019+0000 7f48230c8640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f481c0fc7b0 con 0x7f481c102c90 2024-06-18T03:01:48.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:48.019+0000 7f4820e64640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f481c1022f0 0x7f481c0fbbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:01:48.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:48.019+0000 7f4820e64640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f481c1022f0 0x7f481c0fbbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45050/0 (socket says 172.21.15.16:45050) 2024-06-18T03:01:48.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:48.019+0000 7f4820e64640 1 -- 172.21.15.16:0/3679202044 learned_addr learned my addr 172.21.15.16:0/3679202044 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T03:01:48.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:48.019+0000 7f481bfff640 1 --2- 172.21.15.16:0/3679202044 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f481c102c90 0x7f481c0fc100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:01:48.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:48.019+0000 7f4820e64640 1 -- 172.21.15.16:0/3679202044 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f481c102c90 msgr2=0x7f481c0fc100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:01:48.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:48.019+0000 7f4820e64640 1 --2- 172.21.15.16:0/3679202044 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f481c102c90 0x7f481c0fc100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:01:48.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:48.019+0000 7f4820e64640 1 -- 172.21.15.16:0/3679202044 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f480c005040 con 0x7f481c1022f0 2024-06-18T03:01:48.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:48.019+0000 7f481bfff640 1 --2- 172.21.15.16:0/3679202044 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f481c102c90 0x7f481c0fc100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-18T03:01:48.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:48.019+0000 7f4820e64640 1 --2- 172.21.15.16:0/3679202044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f481c1022f0 0x7f481c0fbbc0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f480c003080 tx=0x7f480c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:01:48.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:48.019+0000 7f48197fa640 1 -- 172.21.15.16:0/3679202044 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f480c038070 con 0x7f481c1022f0 2024-06-18T03:01:48.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:48.019+0000 7f48197fa640 1 -- 172.21.15.16:0/3679202044 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f480c008ee0 con 0x7f481c1022f0 2024-06-18T03:01:48.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:48.019+0000 7f48230c8640 1 -- 172.21.15.16:0/3679202044 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f481c0fc9e0 con 0x7f481c1022f0 2024-06-18T03:01:48.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:48.019+0000 7f48197fa640 1 -- 172.21.15.16:0/3679202044 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f480c033820 con 0x7f481c1022f0 2024-06-18T03:01:48.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:48.019+0000 7f48230c8640 1 -- 172.21.15.16:0/3679202044 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f481c119640 con 0x7f481c1022f0 2024-06-18T03:01:48.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:48.021+0000 7f48230c8640 1 -- 172.21.15.16:0/3679202044 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f481c065510 con 0x7f481c1022f0 2024-06-18T03:01:48.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:48.021+0000 7f48197fa640 1 -- 172.21.15.16:0/3679202044 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f480c030020 con 0x7f481c1022f0 2024-06-18T03:01:48.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:48.021+0000 7f48197fa640 1 --2- 172.21.15.16:0/3679202044 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f4804065430 0x7f48040678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:01:48.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:48.022+0000 7f48197fa640 1 -- 172.21.15.16:0/3679202044 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f480c0a2280 con 0x7f481c1022f0 2024-06-18T03:01:48.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:48.022+0000 7f481bfff640 1 --2- 172.21.15.16:0/3679202044 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f4804065430 0x7f48040678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:01:48.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:48.022+0000 7f481bfff640 1 --2- 172.21.15.16:0/3679202044 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f4804065430 0x7f48040678e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f4810007fd0 tx=0x7f4810009040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:01:48.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:48.026+0000 7f48197fa640 1 -- 172.21.15.16:0/3679202044 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f480c0707b0 con 0x7f481c1022f0 2024-06-18T03:01:48.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:48.235+0000 7f48230c8640 1 -- 172.21.15.16:0/3679202044 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f481c074960 con 0x7f481c1022f0 2024-06-18T03:01:48.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:48.235+0000 7f48197fa640 1 -- 172.21.15.16:0/3679202044 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f480c073dd0 con 0x7f481c1022f0 2024-06-18T03:01:48.236 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T03:01:48.236 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-18T03:01:48.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:48.238+0000 7f48230c8640 1 -- 172.21.15.16:0/3679202044 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f4804065430 msgr2=0x7f48040678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:01:48.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:48.238+0000 7f48230c8640 1 --2- 172.21.15.16:0/3679202044 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f4804065430 0x7f48040678e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f4810007fd0 tx=0x7f4810009040 comp rx=0 tx=0).stop 2024-06-18T03:01:48.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:48.238+0000 7f48230c8640 1 -- 172.21.15.16:0/3679202044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f481c1022f0 msgr2=0x7f481c0fbbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:01:48.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:48.238+0000 7f48230c8640 1 --2- 172.21.15.16:0/3679202044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f481c1022f0 0x7f481c0fbbc0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f480c003080 tx=0x7f480c009da0 comp rx=0 tx=0).stop 2024-06-18T03:01:48.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:48.238+0000 7f48230c8640 1 -- 172.21.15.16:0/3679202044 shutdown_connections 2024-06-18T03:01:48.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:48.238+0000 7f48230c8640 1 --2- 172.21.15.16:0/3679202044 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f4804065430 0x7f48040678e0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:01:48.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:48.238+0000 7f48230c8640 1 --2- 172.21.15.16:0/3679202044 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f481c102c90 0x7f481c0fc100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:01:48.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:48.238+0000 7f48230c8640 1 --2- 172.21.15.16:0/3679202044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f481c1022f0 0x7f481c0fbbc0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:01:48.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:48.238+0000 7f48230c8640 1 -- 172.21.15.16:0/3679202044 >> 172.21.15.16:0/3679202044 conn(0x7f481c070a40 msgr2=0x7f481c1050b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:01:48.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:48.238+0000 7f48230c8640 1 -- 172.21.15.16:0/3679202044 shutdown_connections 2024-06-18T03:01:48.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:48.238+0000 7f48230c8640 1 -- 172.21.15.16:0/3679202044 wait complete. 2024-06-18T03:01:48.785 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:01:48 smithi016 ceph-mon[29214]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:01:48.785 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:01:48 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/3679202044' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:01:48.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:01:48 smithi123 ceph-mon[36358]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:01:48.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:01:48 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/3679202044' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:01:50.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:01:50 smithi016 ceph-mon[29214]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:01:50.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:01:50 smithi123 ceph-mon[36358]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:01:51.810 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T03:01:52.088 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T03:01:52.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:01:52 smithi016 ceph-mon[29214]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:01:52.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:01:52 smithi123 ceph-mon[36358]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:01:53.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:53.634+0000 7f2e1f075640 1 -- 172.21.15.16:0/3323309385 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e18101b60 msgr2=0x7f2e180fb1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:01:53.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:53.634+0000 7f2e1f075640 1 --2- 172.21.15.16:0/3323309385 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e18101b60 0x7f2e180fb1f0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c0048d0 tx=0x7f2e0c009da0 comp rx=0 tx=0).stop 2024-06-18T03:01:53.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:53.634+0000 7f2e1f075640 1 -- 172.21.15.16:0/3323309385 shutdown_connections 2024-06-18T03:01:53.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:53.634+0000 7f2e1f075640 1 --2- 172.21.15.16:0/3323309385 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e18101b60 0x7f2e180fb1f0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:01:53.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:53.634+0000 7f2e1f075640 1 --2- 172.21.15.16:0/3323309385 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e181011c0 0x7f2e18101590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:01:53.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:53.634+0000 7f2e1f075640 1 -- 172.21.15.16:0/3323309385 >> 172.21.15.16:0/3323309385 conn(0x7f2e180f6db0 msgr2=0x7f2e180f91c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:01:53.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:53.634+0000 7f2e1f075640 1 -- 172.21.15.16:0/3323309385 shutdown_connections 2024-06-18T03:01:53.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:53.634+0000 7f2e1f075640 1 -- 172.21.15.16:0/3323309385 wait complete. 2024-06-18T03:01:53.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:53.634+0000 7f2e1f075640 1 Processor -- start 2024-06-18T03:01:53.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:53.634+0000 7f2e1f075640 1 -- start start 2024-06-18T03:01:53.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:53.635+0000 7f2e1f075640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e181011c0 0x7f2e18102040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:01:53.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:53.635+0000 7f2e1f075640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e18101b60 0x7f2e18102580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:01:53.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:53.635+0000 7f2e1f075640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e18102b50 con 0x7f2e181011c0 2024-06-18T03:01:53.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:53.635+0000 7f2e1f075640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e18102cc0 con 0x7f2e18101b60 2024-06-18T03:01:53.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:53.635+0000 7f2e1ce11640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e181011c0 0x7f2e18102040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:01:53.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:53.635+0000 7f2e1ce11640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e181011c0 0x7f2e18102040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47840/0 (socket says 172.21.15.16:47840) 2024-06-18T03:01:53.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:53.635+0000 7f2e1ce11640 1 -- 172.21.15.16:0/633602561 learned_addr learned my addr 172.21.15.16:0/633602561 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T03:01:53.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:53.635+0000 7f2e17fff640 1 --2- 172.21.15.16:0/633602561 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e18101b60 0x7f2e18102580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:01:53.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:53.635+0000 7f2e1ce11640 1 -- 172.21.15.16:0/633602561 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e18101b60 msgr2=0x7f2e18102580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:01:53.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:53.635+0000 7f2e1ce11640 1 --2- 172.21.15.16:0/633602561 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e18101b60 0x7f2e18102580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:01:53.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:53.635+0000 7f2e1ce11640 1 -- 172.21.15.16:0/633602561 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e0c005040 con 0x7f2e181011c0 2024-06-18T03:01:53.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:53.635+0000 7f2e17fff640 1 --2- 172.21.15.16:0/633602561 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e18101b60 0x7f2e18102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-18T03:01:53.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:53.635+0000 7f2e1ce11640 1 --2- 172.21.15.16:0/633602561 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e181011c0 0x7f2e18102040 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f2e0800b790 tx=0x7f2e080091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:01:53.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:53.635+0000 7f2e157fa640 1 -- 172.21.15.16:0/633602561 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e0800c700 con 0x7f2e181011c0 2024-06-18T03:01:53.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:53.635+0000 7f2e1f075640 1 -- 172.21.15.16:0/633602561 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e181193c0 con 0x7f2e181011c0 2024-06-18T03:01:53.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:53.635+0000 7f2e157fa640 1 -- 172.21.15.16:0/633602561 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2e0800cd30 con 0x7f2e181011c0 2024-06-18T03:01:53.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:53.635+0000 7f2e157fa640 1 -- 172.21.15.16:0/633602561 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e08013370 con 0x7f2e181011c0 2024-06-18T03:01:53.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:53.635+0000 7f2e1f075640 1 -- 172.21.15.16:0/633602561 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e18119800 con 0x7f2e181011c0 2024-06-18T03:01:53.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:53.636+0000 7f2e1f075640 1 -- 172.21.15.16:0/633602561 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e180fc930 con 0x7f2e181011c0 2024-06-18T03:01:53.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:53.637+0000 7f2e157fa640 1 -- 172.21.15.16:0/633602561 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f2e08012070 con 0x7f2e181011c0 2024-06-18T03:01:53.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:53.637+0000 7f2e157fa640 1 --2- 172.21.15.16:0/633602561 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f2e00065430 0x7f2e000678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:01:53.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:53.637+0000 7f2e157fa640 1 -- 172.21.15.16:0/633602561 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f2e080820d0 con 0x7f2e181011c0 2024-06-18T03:01:53.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:53.637+0000 7f2e17fff640 1 --2- 172.21.15.16:0/633602561 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f2e00065430 0x7f2e000678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:01:53.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:53.638+0000 7f2e17fff640 1 --2- 172.21.15.16:0/633602561 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f2e00065430 0x7f2e000678e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c009f80 tx=0x7f2e0c009da0 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:01:53.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:53.642+0000 7f2e157fa640 1 -- 172.21.15.16:0/633602561 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f2e08050580 con 0x7f2e181011c0 2024-06-18T03:01:53.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:53.852+0000 7f2e1f075640 1 -- 172.21.15.16:0/633602561 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f2e1804b510 con 0x7f2e181011c0 2024-06-18T03:01:53.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:53.853+0000 7f2e157fa640 1 -- 172.21.15.16:0/633602561 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f2e08053ba0 con 0x7f2e181011c0 2024-06-18T03:01:53.853 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T03:01:53.853 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-18T03:01:53.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:53.855+0000 7f2e1f075640 1 -- 172.21.15.16:0/633602561 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f2e00065430 msgr2=0x7f2e000678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:01:53.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:53.855+0000 7f2e1f075640 1 --2- 172.21.15.16:0/633602561 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f2e00065430 0x7f2e000678e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c009f80 tx=0x7f2e0c009da0 comp rx=0 tx=0).stop 2024-06-18T03:01:53.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:53.855+0000 7f2e1f075640 1 -- 172.21.15.16:0/633602561 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e181011c0 msgr2=0x7f2e18102040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:01:53.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:53.855+0000 7f2e1f075640 1 --2- 172.21.15.16:0/633602561 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e181011c0 0x7f2e18102040 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f2e0800b790 tx=0x7f2e080091d0 comp rx=0 tx=0).stop 2024-06-18T03:01:53.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:53.855+0000 7f2e1f075640 1 -- 172.21.15.16:0/633602561 shutdown_connections 2024-06-18T03:01:53.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:53.855+0000 7f2e1f075640 1 --2- 172.21.15.16:0/633602561 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f2e00065430 0x7f2e000678e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:01:53.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:53.855+0000 7f2e1f075640 1 --2- 172.21.15.16:0/633602561 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e18101b60 0x7f2e18102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:01:53.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:53.855+0000 7f2e1f075640 1 --2- 172.21.15.16:0/633602561 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e181011c0 0x7f2e18102040 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:01:53.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:53.855+0000 7f2e1f075640 1 -- 172.21.15.16:0/633602561 >> 172.21.15.16:0/633602561 conn(0x7f2e180f6db0 msgr2=0x7f2e180f7d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:01:53.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:53.855+0000 7f2e1f075640 1 -- 172.21.15.16:0/633602561 shutdown_connections 2024-06-18T03:01:53.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:53.855+0000 7f2e1f075640 1 -- 172.21.15.16:0/633602561 wait complete. 2024-06-18T03:01:54.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:01:54 smithi016 ceph-mon[29214]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:01:54.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:01:54 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/633602561' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:01:54.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:01:54 smithi123 ceph-mon[36358]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:01:54.985 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:01:54 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/633602561' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:01:56.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:01:56 smithi016 ceph-mon[29214]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:01:56.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:01:56 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T03:01:56.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:01:56 smithi123 ceph-mon[36358]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:01:56.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:01:56 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T03:01:57.517 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T03:01:57.790 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T03:01:58.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:01:58 smithi016 ceph-mon[29214]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:01:58.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:01:58 smithi123 ceph-mon[36358]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:01:59.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:59.346+0000 7f0781a9e640 1 -- 172.21.15.16:0/3502208570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f077c1001d0 msgr2=0x7f077c1005a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:01:59.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:59.346+0000 7f0781a9e640 1 --2- 172.21.15.16:0/3502208570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f077c1001d0 0x7f077c1005a0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f076c0048d0 tx=0x7f076c009da0 comp rx=0 tx=0).stop 2024-06-18T03:01:59.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:59.346+0000 7f0781a9e640 1 -- 172.21.15.16:0/3502208570 shutdown_connections 2024-06-18T03:01:59.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:59.346+0000 7f0781a9e640 1 --2- 172.21.15.16:0/3502208570 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f077c0fbaf0 0x7f077c0fbf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:01:59.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:59.346+0000 7f0781a9e640 1 --2- 172.21.15.16:0/3502208570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f077c1001d0 0x7f077c1005a0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:01:59.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:59.346+0000 7f0781a9e640 1 -- 172.21.15.16:0/3502208570 >> 172.21.15.16:0/3502208570 conn(0x7f077c070a40 msgr2=0x7f077c070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:01:59.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:59.346+0000 7f0781a9e640 1 -- 172.21.15.16:0/3502208570 shutdown_connections 2024-06-18T03:01:59.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:59.346+0000 7f0781a9e640 1 -- 172.21.15.16:0/3502208570 wait complete. 2024-06-18T03:01:59.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:59.347+0000 7f0781a9e640 1 Processor -- start 2024-06-18T03:01:59.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:59.347+0000 7f0781a9e640 1 -- start start 2024-06-18T03:01:59.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:59.347+0000 7f0781a9e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f077c0fbaf0 0x7f077c0fce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:01:59.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:59.347+0000 7f0781a9e640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f077c1001d0 0x7f077c0fd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:01:59.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:59.347+0000 7f0781a9e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f077c0fd990 con 0x7f077c0fbaf0 2024-06-18T03:01:59.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:59.347+0000 7f0781a9e640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f077c0fdb00 con 0x7f077c1001d0 2024-06-18T03:01:59.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:59.347+0000 7f077b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f077c0fbaf0 0x7f077c0fce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:01:59.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:59.347+0000 7f077b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f077c0fbaf0 0x7f077c0fce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47866/0 (socket says 172.21.15.16:47866) 2024-06-18T03:01:59.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:59.347+0000 7f077b7fe640 1 -- 172.21.15.16:0/1277702843 learned_addr learned my addr 172.21.15.16:0/1277702843 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T03:01:59.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:59.347+0000 7f077affd640 1 --2- 172.21.15.16:0/1277702843 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f077c1001d0 0x7f077c0fd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:01:59.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:59.348+0000 7f077b7fe640 1 -- 172.21.15.16:0/1277702843 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f077c1001d0 msgr2=0x7f077c0fd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:01:59.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:59.348+0000 7f077b7fe640 1 --2- 172.21.15.16:0/1277702843 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f077c1001d0 0x7f077c0fd3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:01:59.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:59.348+0000 7f077b7fe640 1 -- 172.21.15.16:0/1277702843 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f076c005040 con 0x7f077c0fbaf0 2024-06-18T03:01:59.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:59.348+0000 7f077affd640 1 --2- 172.21.15.16:0/1277702843 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f077c1001d0 0x7f077c0fd3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-18T03:01:59.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:59.348+0000 7f077b7fe640 1 --2- 172.21.15.16:0/1277702843 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f077c0fbaf0 0x7f077c0fce80 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f076c003080 tx=0x7f076c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:01:59.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:59.348+0000 7f0763fff640 1 -- 172.21.15.16:0/1277702843 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f076c038070 con 0x7f077c0fbaf0 2024-06-18T03:01:59.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:59.348+0000 7f0781a9e640 1 -- 172.21.15.16:0/1277702843 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f077c119430 con 0x7f077c0fbaf0 2024-06-18T03:01:59.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:59.348+0000 7f0763fff640 1 -- 172.21.15.16:0/1277702843 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f076c008ee0 con 0x7f077c0fbaf0 2024-06-18T03:01:59.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:59.348+0000 7f0763fff640 1 -- 172.21.15.16:0/1277702843 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f076c03d870 con 0x7f077c0fbaf0 2024-06-18T03:01:59.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:59.348+0000 7f0781a9e640 1 -- 172.21.15.16:0/1277702843 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f077c1197b0 con 0x7f077c0fbaf0 2024-06-18T03:01:59.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:59.349+0000 7f0781a9e640 1 -- 172.21.15.16:0/1277702843 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f077c0fcb00 con 0x7f077c0fbaf0 2024-06-18T03:01:59.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:59.350+0000 7f0763fff640 1 -- 172.21.15.16:0/1277702843 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f076c030020 con 0x7f077c0fbaf0 2024-06-18T03:01:59.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:59.350+0000 7f0763fff640 1 --2- 172.21.15.16:0/1277702843 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f0764065430 0x7f07640678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:01:59.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:59.350+0000 7f0763fff640 1 -- 172.21.15.16:0/1277702843 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f076c0a21d0 con 0x7f077c0fbaf0 2024-06-18T03:01:59.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:59.350+0000 7f077affd640 1 --2- 172.21.15.16:0/1277702843 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f0764065430 0x7f07640678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:01:59.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:59.351+0000 7f077affd640 1 --2- 172.21.15.16:0/1277702843 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f0764065430 0x7f07640678e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f077c0fe7c0 tx=0x7f0770008040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:01:59.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:59.355+0000 7f0763fff640 1 -- 172.21.15.16:0/1277702843 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f076c070700 con 0x7f077c0fbaf0 2024-06-18T03:01:59.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:59.566+0000 7f0781a9e640 1 -- 172.21.15.16:0/1277702843 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f077c074960 con 0x7f077c0fbaf0 2024-06-18T03:01:59.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:59.567+0000 7f0763fff640 1 -- 172.21.15.16:0/1277702843 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f076c073d20 con 0x7f077c0fbaf0 2024-06-18T03:01:59.567 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T03:01:59.567 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-18T03:01:59.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:59.568+0000 7f0781a9e640 1 -- 172.21.15.16:0/1277702843 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f0764065430 msgr2=0x7f07640678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:01:59.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:59.568+0000 7f0781a9e640 1 --2- 172.21.15.16:0/1277702843 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f0764065430 0x7f07640678e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f077c0fe7c0 tx=0x7f0770008040 comp rx=0 tx=0).stop 2024-06-18T03:01:59.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:59.568+0000 7f0781a9e640 1 -- 172.21.15.16:0/1277702843 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f077c0fbaf0 msgr2=0x7f077c0fce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:01:59.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:59.568+0000 7f0781a9e640 1 --2- 172.21.15.16:0/1277702843 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f077c0fbaf0 0x7f077c0fce80 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f076c003080 tx=0x7f076c009da0 comp rx=0 tx=0).stop 2024-06-18T03:01:59.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:59.569+0000 7f0781a9e640 1 -- 172.21.15.16:0/1277702843 shutdown_connections 2024-06-18T03:01:59.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:59.569+0000 7f0781a9e640 1 --2- 172.21.15.16:0/1277702843 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f0764065430 0x7f07640678e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:01:59.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:59.569+0000 7f0781a9e640 1 --2- 172.21.15.16:0/1277702843 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f077c1001d0 0x7f077c0fd3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:01:59.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:59.569+0000 7f0781a9e640 1 --2- 172.21.15.16:0/1277702843 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f077c0fbaf0 0x7f077c0fce80 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:01:59.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:59.569+0000 7f0781a9e640 1 -- 172.21.15.16:0/1277702843 >> 172.21.15.16:0/1277702843 conn(0x7f077c070a40 msgr2=0x7f077c0fb270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:01:59.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:59.569+0000 7f0781a9e640 1 -- 172.21.15.16:0/1277702843 shutdown_connections 2024-06-18T03:01:59.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:01:59.569+0000 7f0781a9e640 1 -- 172.21.15.16:0/1277702843 wait complete. 2024-06-18T03:02:00.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:02:00 smithi016 ceph-mon[29214]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:02:00.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:02:00 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/1277702843' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:02:00.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:02:00 smithi123 ceph-mon[36358]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:02:00.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:02:00 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/1277702843' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:02:02.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:02:02 smithi016 ceph-mon[29214]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:02:02.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:02:02 smithi123 ceph-mon[36358]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:02:03.224 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T03:02:03.503 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T03:02:04.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:02:04 smithi016 ceph-mon[29214]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:02:04.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:02:04 smithi123 ceph-mon[36358]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:02:05.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:05.042+0000 7fb1a7850640 1 -- 172.21.15.16:0/1344012977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1a0102c60 msgr2=0x7fb1a00fc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:02:05.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:05.042+0000 7fb1a7850640 1 --2- 172.21.15.16:0/1344012977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1a0102c60 0x7fb1a00fc2f0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fb1900048d0 tx=0x7fb190009da0 comp rx=0 tx=0).stop 2024-06-18T03:02:05.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:05.042+0000 7fb1a7850640 1 -- 172.21.15.16:0/1344012977 shutdown_connections 2024-06-18T03:02:05.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:05.042+0000 7fb1a7850640 1 --2- 172.21.15.16:0/1344012977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1a0102c60 0x7fb1a00fc2f0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:02:05.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:05.042+0000 7fb1a7850640 1 --2- 172.21.15.16:0/1344012977 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1a01022c0 0x7fb1a0102690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:02:05.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:05.042+0000 7fb1a7850640 1 -- 172.21.15.16:0/1344012977 >> 172.21.15.16:0/1344012977 conn(0x7fb1a00f7e50 msgr2=0x7fb1a00fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:02:05.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:05.043+0000 7fb1a7850640 1 -- 172.21.15.16:0/1344012977 shutdown_connections 2024-06-18T03:02:05.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:05.043+0000 7fb1a7850640 1 -- 172.21.15.16:0/1344012977 wait complete. 2024-06-18T03:02:05.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:05.043+0000 7fb1a7850640 1 Processor -- start 2024-06-18T03:02:05.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:05.043+0000 7fb1a7850640 1 -- start start 2024-06-18T03:02:05.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:05.043+0000 7fb1a7850640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1a01022c0 0x7fb1a006f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:02:05.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:05.043+0000 7fb1a7850640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1a0102c60 0x7fb1a006fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:02:05.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:05.043+0000 7fb1a7850640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1a00701f0 con 0x7fb1a0102c60 2024-06-18T03:02:05.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:05.043+0000 7fb1a7850640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1a0070360 con 0x7fb1a01022c0 2024-06-18T03:02:05.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:05.043+0000 7fb1a4deb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1a0102c60 0x7fb1a006fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:02:05.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:05.043+0000 7fb1a4deb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1a0102c60 0x7fb1a006fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53396/0 (socket says 172.21.15.16:53396) 2024-06-18T03:02:05.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:05.043+0000 7fb1a4deb640 1 -- 172.21.15.16:0/2119048507 learned_addr learned my addr 172.21.15.16:0/2119048507 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T03:02:05.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:05.044+0000 7fb1a55ec640 1 --2- 172.21.15.16:0/2119048507 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1a01022c0 0x7fb1a006f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:02:05.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:05.044+0000 7fb1a4deb640 1 -- 172.21.15.16:0/2119048507 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1a01022c0 msgr2=0x7fb1a006f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:02:05.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:05.044+0000 7fb1a4deb640 1 --2- 172.21.15.16:0/2119048507 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1a01022c0 0x7fb1a006f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:02:05.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:05.044+0000 7fb1a4deb640 1 -- 172.21.15.16:0/2119048507 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb190005040 con 0x7fb1a0102c60 2024-06-18T03:02:05.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:05.044+0000 7fb1a4deb640 1 --2- 172.21.15.16:0/2119048507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1a0102c60 0x7fb1a006fcb0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fb1900043d0 tx=0x7fb190009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:02:05.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:05.044+0000 7fb195ffb640 1 -- 172.21.15.16:0/2119048507 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb190038070 con 0x7fb1a0102c60 2024-06-18T03:02:05.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:05.044+0000 7fb195ffb640 1 -- 172.21.15.16:0/2119048507 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb190032970 con 0x7fb1a0102c60 2024-06-18T03:02:05.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:05.044+0000 7fb1a7850640 1 -- 172.21.15.16:0/2119048507 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1a0070590 con 0x7fb1a0102c60 2024-06-18T03:02:05.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:05.044+0000 7fb195ffb640 1 -- 172.21.15.16:0/2119048507 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb19003da60 con 0x7fb1a0102c60 2024-06-18T03:02:05.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:05.044+0000 7fb1a7850640 1 -- 172.21.15.16:0/2119048507 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1a01199f0 con 0x7fb1a0102c60 2024-06-18T03:02:05.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:05.045+0000 7fb1a7850640 1 -- 172.21.15.16:0/2119048507 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1a00fda30 con 0x7fb1a0102c60 2024-06-18T03:02:05.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:05.046+0000 7fb195ffb640 1 -- 172.21.15.16:0/2119048507 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fb190030020 con 0x7fb1a0102c60 2024-06-18T03:02:05.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:05.046+0000 7fb195ffb640 1 --2- 172.21.15.16:0/2119048507 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fb18c065430 0x7fb18c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:02:05.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:05.046+0000 7fb195ffb640 1 -- 172.21.15.16:0/2119048507 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fb1900a2590 con 0x7fb1a0102c60 2024-06-18T03:02:05.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:05.046+0000 7fb1a55ec640 1 --2- 172.21.15.16:0/2119048507 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fb18c065430 0x7fb18c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:02:05.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:05.047+0000 7fb1a55ec640 1 --2- 172.21.15.16:0/2119048507 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fb18c065430 0x7fb18c0678e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb19c007fd0 tx=0x7fb19c009e90 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:02:05.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:05.050+0000 7fb195ffb640 1 -- 172.21.15.16:0/2119048507 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fb190070a40 con 0x7fb1a0102c60 2024-06-18T03:02:05.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:05.258+0000 7fb1a7850640 1 -- 172.21.15.16:0/2119048507 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fb1a00fbab0 con 0x7fb1a0102c60 2024-06-18T03:02:05.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:05.258+0000 7fb195ffb640 1 -- 172.21.15.16:0/2119048507 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fb190074060 con 0x7fb1a0102c60 2024-06-18T03:02:05.259 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T03:02:05.259 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-18T03:02:05.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:05.261+0000 7fb1a7850640 1 -- 172.21.15.16:0/2119048507 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fb18c065430 msgr2=0x7fb18c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:02:05.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:05.261+0000 7fb1a7850640 1 --2- 172.21.15.16:0/2119048507 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fb18c065430 0x7fb18c0678e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb19c007fd0 tx=0x7fb19c009e90 comp rx=0 tx=0).stop 2024-06-18T03:02:05.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:05.261+0000 7fb1a7850640 1 -- 172.21.15.16:0/2119048507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1a0102c60 msgr2=0x7fb1a006fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:02:05.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:05.261+0000 7fb1a7850640 1 --2- 172.21.15.16:0/2119048507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1a0102c60 0x7fb1a006fcb0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fb1900043d0 tx=0x7fb190009da0 comp rx=0 tx=0).stop 2024-06-18T03:02:05.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:05.261+0000 7fb1a7850640 1 -- 172.21.15.16:0/2119048507 shutdown_connections 2024-06-18T03:02:05.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:05.261+0000 7fb1a7850640 1 --2- 172.21.15.16:0/2119048507 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fb18c065430 0x7fb18c0678e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:02:05.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:05.261+0000 7fb1a7850640 1 --2- 172.21.15.16:0/2119048507 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb1a0102c60 0x7fb1a006fcb0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:02:05.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:05.261+0000 7fb1a7850640 1 --2- 172.21.15.16:0/2119048507 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1a01022c0 0x7fb1a006f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:02:05.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:05.261+0000 7fb1a7850640 1 -- 172.21.15.16:0/2119048507 >> 172.21.15.16:0/2119048507 conn(0x7fb1a00f7e50 msgr2=0x7fb1a00f8db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:02:05.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:05.261+0000 7fb1a7850640 1 -- 172.21.15.16:0/2119048507 shutdown_connections 2024-06-18T03:02:05.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:05.261+0000 7fb1a7850640 1 -- 172.21.15.16:0/2119048507 wait complete. 2024-06-18T03:02:05.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:02:05 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/2119048507' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:02:05.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:02:05 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/2119048507' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:02:06.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:02:06 smithi123 ceph-mon[36358]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:02:07.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:02:06 smithi016 ceph-mon[29214]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:02:08.889 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T03:02:08.919 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:02:08 smithi016 ceph-mon[29214]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:02:08.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:02:08 smithi123 ceph-mon[36358]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:02:09.166 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T03:02:10.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:10.740+0000 7f9df2c14640 1 -- 172.21.15.16:0/692977647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9dec0faa50 msgr2=0x7f9dec0faec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:02:10.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:10.740+0000 7f9df2c14640 1 --2- 172.21.15.16:0/692977647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9dec0faa50 0x7f9dec0faec0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f9dd40048d0 tx=0x7f9dd4009da0 comp rx=0 tx=0).stop 2024-06-18T03:02:10.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:10.740+0000 7f9df2c14640 1 -- 172.21.15.16:0/692977647 shutdown_connections 2024-06-18T03:02:10.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:10.740+0000 7f9df2c14640 1 --2- 172.21.15.16:0/692977647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9dec0faa50 0x7f9dec0faec0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:02:10.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:10.740+0000 7f9df2c14640 1 --2- 172.21.15.16:0/692977647 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9dec1023e0 0x7f9dec1027b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:02:10.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:10.740+0000 7f9df2c14640 1 -- 172.21.15.16:0/692977647 >> 172.21.15.16:0/692977647 conn(0x7f9dec070ab0 msgr2=0x7f9dec070eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:02:10.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:10.740+0000 7f9df2c14640 1 -- 172.21.15.16:0/692977647 shutdown_connections 2024-06-18T03:02:10.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:10.740+0000 7f9df2c14640 1 -- 172.21.15.16:0/692977647 wait complete. 2024-06-18T03:02:10.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:10.741+0000 7f9df2c14640 1 Processor -- start 2024-06-18T03:02:10.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:10.741+0000 7f9df2c14640 1 -- start start 2024-06-18T03:02:10.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:10.741+0000 7f9df2c14640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9dec0faa50 0x7f9dec1130e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:02:10.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:10.741+0000 7f9df2c14640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9dec1023e0 0x7f9dec113620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:02:10.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:10.741+0000 7f9df2c14640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dec113c20 con 0x7f9dec0faa50 2024-06-18T03:02:10.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:10.741+0000 7f9df2c14640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dec10d060 con 0x7f9dec1023e0 2024-06-18T03:02:10.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:10.741+0000 7f9df09b0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9dec0faa50 0x7f9dec1130e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:02:10.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:10.741+0000 7f9df09b0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9dec0faa50 0x7f9dec1130e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53414/0 (socket says 172.21.15.16:53414) 2024-06-18T03:02:10.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:10.741+0000 7f9df09b0640 1 -- 172.21.15.16:0/3129666210 learned_addr learned my addr 172.21.15.16:0/3129666210 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T03:02:10.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:10.742+0000 7f9debfff640 1 --2- 172.21.15.16:0/3129666210 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9dec1023e0 0x7f9dec113620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:02:10.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:10.742+0000 7f9df09b0640 1 -- 172.21.15.16:0/3129666210 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9dec1023e0 msgr2=0x7f9dec113620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:02:10.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:10.742+0000 7f9df09b0640 1 --2- 172.21.15.16:0/3129666210 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9dec1023e0 0x7f9dec113620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:02:10.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:10.742+0000 7f9df09b0640 1 -- 172.21.15.16:0/3129666210 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dd4005040 con 0x7f9dec0faa50 2024-06-18T03:02:10.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:10.742+0000 7f9df09b0640 1 --2- 172.21.15.16:0/3129666210 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9dec0faa50 0x7f9dec1130e0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f9ddc002a50 tx=0x7f9ddc004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:02:10.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:10.742+0000 7f9de97fa640 1 -- 172.21.15.16:0/3129666210 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ddc007c30 con 0x7f9dec0faa50 2024-06-18T03:02:10.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:10.742+0000 7f9df2c14640 1 -- 172.21.15.16:0/3129666210 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dec10d340 con 0x7f9dec0faa50 2024-06-18T03:02:10.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:10.742+0000 7f9de97fa640 1 -- 172.21.15.16:0/3129666210 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9ddc00f040 con 0x7f9dec0faa50 2024-06-18T03:02:10.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:10.742+0000 7f9de97fa640 1 -- 172.21.15.16:0/3129666210 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ddc00db50 con 0x7f9dec0faa50 2024-06-18T03:02:10.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:10.742+0000 7f9df2c14640 1 -- 172.21.15.16:0/3129666210 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dec10d860 con 0x7f9dec0faa50 2024-06-18T03:02:10.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:10.743+0000 7f9df2c14640 1 -- 172.21.15.16:0/3129666210 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dec065550 con 0x7f9dec0faa50 2024-06-18T03:02:10.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:10.744+0000 7f9de97fa640 1 -- 172.21.15.16:0/3129666210 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f9ddc01c400 con 0x7f9dec0faa50 2024-06-18T03:02:10.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:10.744+0000 7f9de97fa640 1 --2- 172.21.15.16:0/3129666210 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f9dd8065430 0x7f9dd80678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:02:10.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:10.744+0000 7f9de97fa640 1 -- 172.21.15.16:0/3129666210 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f9ddc081e00 con 0x7f9dec0faa50 2024-06-18T03:02:10.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:10.744+0000 7f9debfff640 1 --2- 172.21.15.16:0/3129666210 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f9dd8065430 0x7f9dd80678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:02:10.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:10.745+0000 7f9debfff640 1 --2- 172.21.15.16:0/3129666210 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f9dd8065430 0x7f9dd80678e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9dd4000c00 tx=0x7f9dd4009da0 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:02:10.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:10.748+0000 7f9de97fa640 1 -- 172.21.15.16:0/3129666210 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f9ddc0502b0 con 0x7f9dec0faa50 2024-06-18T03:02:10.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:02:10 smithi016 ceph-mon[29214]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:02:10.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:10.955+0000 7f9df2c14640 1 -- 172.21.15.16:0/3129666210 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f9dec04b480 con 0x7f9dec0faa50 2024-06-18T03:02:10.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:10.956+0000 7f9de97fa640 1 -- 172.21.15.16:0/3129666210 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f9ddc0538d0 con 0x7f9dec0faa50 2024-06-18T03:02:10.956 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T03:02:10.956 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-18T03:02:10.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:10.958+0000 7f9df2c14640 1 -- 172.21.15.16:0/3129666210 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f9dd8065430 msgr2=0x7f9dd80678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:02:10.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:10.958+0000 7f9df2c14640 1 --2- 172.21.15.16:0/3129666210 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f9dd8065430 0x7f9dd80678e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9dd4000c00 tx=0x7f9dd4009da0 comp rx=0 tx=0).stop 2024-06-18T03:02:10.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:10.958+0000 7f9df2c14640 1 -- 172.21.15.16:0/3129666210 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9dec0faa50 msgr2=0x7f9dec1130e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:02:10.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:10.958+0000 7f9df2c14640 1 --2- 172.21.15.16:0/3129666210 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9dec0faa50 0x7f9dec1130e0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f9ddc002a50 tx=0x7f9ddc004600 comp rx=0 tx=0).stop 2024-06-18T03:02:10.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:10.958+0000 7f9df2c14640 1 -- 172.21.15.16:0/3129666210 shutdown_connections 2024-06-18T03:02:10.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:10.958+0000 7f9df2c14640 1 --2- 172.21.15.16:0/3129666210 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f9dd8065430 0x7f9dd80678e0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:02:10.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:10.958+0000 7f9df2c14640 1 --2- 172.21.15.16:0/3129666210 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9dec1023e0 0x7f9dec113620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:02:10.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:10.958+0000 7f9df2c14640 1 --2- 172.21.15.16:0/3129666210 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9dec0faa50 0x7f9dec1130e0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:02:10.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:10.958+0000 7f9df2c14640 1 -- 172.21.15.16:0/3129666210 >> 172.21.15.16:0/3129666210 conn(0x7f9dec070ab0 msgr2=0x7f9dec0f9f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:02:10.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:10.958+0000 7f9df2c14640 1 -- 172.21.15.16:0/3129666210 shutdown_connections 2024-06-18T03:02:10.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:10.958+0000 7f9df2c14640 1 -- 172.21.15.16:0/3129666210 wait complete. 2024-06-18T03:02:10.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:02:10 smithi123 ceph-mon[36358]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:02:11.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:02:11 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/3129666210' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:02:11.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:02:11 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T03:02:11.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:02:11 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/3129666210' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:02:11.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:02:11 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T03:02:12.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:02:12 smithi123 ceph-mon[36358]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:02:12.992 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:02:12 smithi016 ceph-mon[29214]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:02:14.555 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T03:02:14.826 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T03:02:14.949 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:02:14 smithi016 ceph-mon[29214]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:02:14.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:02:14 smithi123 ceph-mon[36358]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:02:16.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:16.261+0000 7f69afb00640 1 -- 172.21.15.16:0/3884168168 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69a8070120 msgr2=0x7f69a8108620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:02:16.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:16.261+0000 7f69afb00640 1 --2- 172.21.15.16:0/3884168168 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69a8070120 0x7f69a8108620 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f69980048d0 tx=0x7f6998009da0 comp rx=0 tx=0).stop 2024-06-18T03:02:16.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:16.261+0000 7f69afb00640 1 -- 172.21.15.16:0/3884168168 shutdown_connections 2024-06-18T03:02:16.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:16.261+0000 7f69afb00640 1 --2- 172.21.15.16:0/3884168168 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69a8070120 0x7f69a8108620 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:02:16.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:16.261+0000 7f69afb00640 1 --2- 172.21.15.16:0/3884168168 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69a806f810 0x7f69a806fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:02:16.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:16.261+0000 7f69afb00640 1 -- 172.21.15.16:0/3884168168 >> 172.21.15.16:0/3884168168 conn(0x7f69a80f7f00 msgr2=0x7f69a80fa310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:02:16.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:16.261+0000 7f69afb00640 1 -- 172.21.15.16:0/3884168168 shutdown_connections 2024-06-18T03:02:16.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:16.261+0000 7f69afb00640 1 -- 172.21.15.16:0/3884168168 wait complete. 2024-06-18T03:02:16.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:16.262+0000 7f69afb00640 1 Processor -- start 2024-06-18T03:02:16.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:16.262+0000 7f69afb00640 1 -- start start 2024-06-18T03:02:16.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:16.262+0000 7f69afb00640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69a806f810 0x7f69a8115270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:02:16.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:16.262+0000 7f69afb00640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69a8070120 0x7f69a81157b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:02:16.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:16.262+0000 7f69afb00640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a8115db0 con 0x7f69a806f810 2024-06-18T03:02:16.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:16.262+0000 7f69afb00640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a810f1f0 con 0x7f69a8070120 2024-06-18T03:02:16.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:16.262+0000 7f69ad89c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69a806f810 0x7f69a8115270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:02:16.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:16.262+0000 7f69ad89c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69a806f810 0x7f69a8115270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58654/0 (socket says 172.21.15.16:58654) 2024-06-18T03:02:16.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:16.262+0000 7f69ad89c640 1 -- 172.21.15.16:0/4263550878 learned_addr learned my addr 172.21.15.16:0/4263550878 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T03:02:16.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:16.262+0000 7f69ad09b640 1 --2- 172.21.15.16:0/4263550878 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69a8070120 0x7f69a81157b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:02:16.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:16.262+0000 7f69ad89c640 1 -- 172.21.15.16:0/4263550878 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69a8070120 msgr2=0x7f69a81157b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:02:16.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:16.262+0000 7f69ad89c640 1 --2- 172.21.15.16:0/4263550878 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69a8070120 0x7f69a81157b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:02:16.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:16.262+0000 7f69ad89c640 1 -- 172.21.15.16:0/4263550878 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6998005040 con 0x7f69a806f810 2024-06-18T03:02:16.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:16.262+0000 7f69ad09b640 1 --2- 172.21.15.16:0/4263550878 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69a8070120 0x7f69a81157b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-18T03:02:16.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:16.263+0000 7f69ad89c640 1 --2- 172.21.15.16:0/4263550878 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69a806f810 0x7f69a8115270 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f69a400c830 tx=0x7f69a4004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:02:16.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:16.263+0000 7f699e7fc640 1 -- 172.21.15.16:0/4263550878 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f69a400a850 con 0x7f69a806f810 2024-06-18T03:02:16.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:16.263+0000 7f69afb00640 1 -- 172.21.15.16:0/4263550878 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69a810f4d0 con 0x7f69a806f810 2024-06-18T03:02:16.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:16.263+0000 7f699e7fc640 1 -- 172.21.15.16:0/4263550878 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f69a400ae80 con 0x7f69a806f810 2024-06-18T03:02:16.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:16.263+0000 7f699e7fc640 1 -- 172.21.15.16:0/4263550878 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f69a4012ea0 con 0x7f69a806f810 2024-06-18T03:02:16.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:16.263+0000 7f69afb00640 1 -- 172.21.15.16:0/4263550878 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69a810f9c0 con 0x7f69a806f810 2024-06-18T03:02:16.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:16.264+0000 7f69afb00640 1 -- 172.21.15.16:0/4263550878 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69a8070f50 con 0x7f69a806f810 2024-06-18T03:02:16.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:16.264+0000 7f699e7fc640 1 -- 172.21.15.16:0/4263550878 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f69a4012400 con 0x7f69a806f810 2024-06-18T03:02:16.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:16.265+0000 7f699e7fc640 1 --2- 172.21.15.16:0/4263550878 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f6994065430 0x7f69940678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:02:16.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:16.265+0000 7f699e7fc640 1 -- 172.21.15.16:0/4263550878 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f69a40827d0 con 0x7f69a806f810 2024-06-18T03:02:16.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:16.265+0000 7f69ad09b640 1 --2- 172.21.15.16:0/4263550878 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f6994065430 0x7f69940678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:02:16.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:16.265+0000 7f69ad09b640 1 --2- 172.21.15.16:0/4263550878 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f6994065430 0x7f69940678e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f69a81009f0 tx=0x7f6998009da0 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:02:16.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:16.269+0000 7f699e7fc640 1 -- 172.21.15.16:0/4263550878 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f69a4050c60 con 0x7f69a806f810 2024-06-18T03:02:16.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:16.483+0000 7f69afb00640 1 -- 172.21.15.16:0/4263550878 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f69a80fbb00 con 0x7f69a806f810 2024-06-18T03:02:16.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:16.484+0000 7f699e7fc640 1 -- 172.21.15.16:0/4263550878 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f69a40126b0 con 0x7f69a806f810 2024-06-18T03:02:16.484 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T03:02:16.484 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-18T03:02:16.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:16.486+0000 7f69afb00640 1 -- 172.21.15.16:0/4263550878 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f6994065430 msgr2=0x7f69940678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:02:16.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:16.486+0000 7f69afb00640 1 --2- 172.21.15.16:0/4263550878 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f6994065430 0x7f69940678e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f69a81009f0 tx=0x7f6998009da0 comp rx=0 tx=0).stop 2024-06-18T03:02:16.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:16.486+0000 7f69afb00640 1 -- 172.21.15.16:0/4263550878 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69a806f810 msgr2=0x7f69a8115270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:02:16.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:16.486+0000 7f69afb00640 1 --2- 172.21.15.16:0/4263550878 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69a806f810 0x7f69a8115270 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f69a400c830 tx=0x7f69a4004430 comp rx=0 tx=0).stop 2024-06-18T03:02:16.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:16.486+0000 7f69afb00640 1 -- 172.21.15.16:0/4263550878 shutdown_connections 2024-06-18T03:02:16.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:16.486+0000 7f69afb00640 1 --2- 172.21.15.16:0/4263550878 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f6994065430 0x7f69940678e0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:02:16.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:16.486+0000 7f69afb00640 1 --2- 172.21.15.16:0/4263550878 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69a8070120 0x7f69a81157b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:02:16.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:16.486+0000 7f69afb00640 1 --2- 172.21.15.16:0/4263550878 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69a806f810 0x7f69a8115270 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:02:16.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:16.486+0000 7f69afb00640 1 -- 172.21.15.16:0/4263550878 >> 172.21.15.16:0/4263550878 conn(0x7f69a80f7f00 msgr2=0x7f69a80f93b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:02:16.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:16.486+0000 7f69afb00640 1 -- 172.21.15.16:0/4263550878 shutdown_connections 2024-06-18T03:02:16.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:16.486+0000 7f69afb00640 1 -- 172.21.15.16:0/4263550878 wait complete. 2024-06-18T03:02:16.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:02:16 smithi016 ceph-mon[29214]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:02:16.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:02:16 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/4263550878' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:02:16.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:02:16 smithi123 ceph-mon[36358]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:02:16.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:02:16 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/4263550878' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:02:18.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:02:18 smithi123 ceph-mon[36358]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:02:19.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:02:18 smithi016 ceph-mon[29214]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:02:19.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:02:19 smithi123 ceph-mon[36358]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:02:20.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:02:19 smithi016 ceph-mon[29214]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:02:20.150 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T03:02:20.430 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T03:02:22.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:22.043+0000 7f0d9ccb0640 1 -- 172.21.15.16:0/3422336174 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d98102c60 msgr2=0x7f0d980fc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:02:22.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:22.043+0000 7f0d9ccb0640 1 --2- 172.21.15.16:0/3422336174 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d98102c60 0x7f0d980fc2f0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f0d800048d0 tx=0x7f0d80009da0 comp rx=0 tx=0).stop 2024-06-18T03:02:22.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:22.043+0000 7f0d9ccb0640 1 -- 172.21.15.16:0/3422336174 shutdown_connections 2024-06-18T03:02:22.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:22.043+0000 7f0d9ccb0640 1 --2- 172.21.15.16:0/3422336174 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d98102c60 0x7f0d980fc2f0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:02:22.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:22.043+0000 7f0d9ccb0640 1 --2- 172.21.15.16:0/3422336174 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d981022c0 0x7f0d98102690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:02:22.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:22.043+0000 7f0d9ccb0640 1 -- 172.21.15.16:0/3422336174 >> 172.21.15.16:0/3422336174 conn(0x7f0d980f7e50 msgr2=0x7f0d980fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:02:22.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:22.043+0000 7f0d9ccb0640 1 -- 172.21.15.16:0/3422336174 shutdown_connections 2024-06-18T03:02:22.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:22.043+0000 7f0d9ccb0640 1 -- 172.21.15.16:0/3422336174 wait complete. 2024-06-18T03:02:22.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:22.044+0000 7f0d9ccb0640 1 Processor -- start 2024-06-18T03:02:22.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:22.044+0000 7f0d9ccb0640 1 -- start start 2024-06-18T03:02:22.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:22.044+0000 7f0d9ccb0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d981022c0 0x7f0d98113370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:02:22.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:22.044+0000 7f0d9ccb0640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d98102c60 0x7f0d981138b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:02:22.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:22.044+0000 7f0d9ccb0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d98113eb0 con 0x7f0d981022c0 2024-06-18T03:02:22.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:22.044+0000 7f0d9ccb0640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d9810d2f0 con 0x7f0d98102c60 2024-06-18T03:02:22.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:22.044+0000 7f0d9659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d981022c0 0x7f0d98113370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:02:22.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:22.044+0000 7f0d9659c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d981022c0 0x7f0d98113370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58624/0 (socket says 172.21.15.16:58624) 2024-06-18T03:02:22.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:22.044+0000 7f0d9659c640 1 -- 172.21.15.16:0/746730391 learned_addr learned my addr 172.21.15.16:0/746730391 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T03:02:22.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:22.044+0000 7f0d95d9b640 1 --2- 172.21.15.16:0/746730391 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d98102c60 0x7f0d981138b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:02:22.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:22.044+0000 7f0d9659c640 1 -- 172.21.15.16:0/746730391 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d98102c60 msgr2=0x7f0d981138b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:02:22.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:22.044+0000 7f0d9659c640 1 --2- 172.21.15.16:0/746730391 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d98102c60 0x7f0d981138b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:02:22.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:22.044+0000 7f0d9659c640 1 -- 172.21.15.16:0/746730391 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d80005040 con 0x7f0d981022c0 2024-06-18T03:02:22.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:22.044+0000 7f0d95d9b640 1 --2- 172.21.15.16:0/746730391 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d98102c60 0x7f0d981138b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-18T03:02:22.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:22.045+0000 7f0d9659c640 1 --2- 172.21.15.16:0/746730391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d981022c0 0x7f0d98113370 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f0d88002a50 tx=0x7f0d88004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:02:22.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:22.045+0000 7f0d8effd640 1 -- 172.21.15.16:0/746730391 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d88007c30 con 0x7f0d981022c0 2024-06-18T03:02:22.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:22.045+0000 7f0d8effd640 1 -- 172.21.15.16:0/746730391 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0d8800f040 con 0x7f0d981022c0 2024-06-18T03:02:22.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:22.045+0000 7f0d9ccb0640 1 -- 172.21.15.16:0/746730391 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d9810d5d0 con 0x7f0d981022c0 2024-06-18T03:02:22.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:22.045+0000 7f0d8effd640 1 -- 172.21.15.16:0/746730391 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d8800db50 con 0x7f0d981022c0 2024-06-18T03:02:22.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:22.045+0000 7f0d9ccb0640 1 -- 172.21.15.16:0/746730391 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d9810db70 con 0x7f0d981022c0 2024-06-18T03:02:22.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:22.047+0000 7f0d9ccb0640 1 -- 172.21.15.16:0/746730391 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d980fda30 con 0x7f0d981022c0 2024-06-18T03:02:22.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:22.047+0000 7f0d8effd640 1 -- 172.21.15.16:0/746730391 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f0d8801c400 con 0x7f0d981022c0 2024-06-18T03:02:22.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:22.047+0000 7f0d8effd640 1 --2- 172.21.15.16:0/746730391 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f0d84065430 0x7f0d840678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:02:22.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:22.048+0000 7f0d8effd640 1 -- 172.21.15.16:0/746730391 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f0d88081ea0 con 0x7f0d981022c0 2024-06-18T03:02:22.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:22.048+0000 7f0d95d9b640 1 --2- 172.21.15.16:0/746730391 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f0d84065430 0x7f0d840678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:02:22.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:22.048+0000 7f0d95d9b640 1 --2- 172.21.15.16:0/746730391 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f0d84065430 0x7f0d840678e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f0d9810ec30 tx=0x7f0d80035040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:02:22.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:22.052+0000 7f0d8effd640 1 -- 172.21.15.16:0/746730391 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f0d880502b0 con 0x7f0d981022c0 2024-06-18T03:02:22.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:22.264+0000 7f0d9ccb0640 1 -- 172.21.15.16:0/746730391 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f0d98109420 con 0x7f0d981022c0 2024-06-18T03:02:22.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:22.264+0000 7f0d8effd640 1 -- 172.21.15.16:0/746730391 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f0d880538d0 con 0x7f0d981022c0 2024-06-18T03:02:22.265 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T03:02:22.265 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-18T03:02:22.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:22.266+0000 7f0d9ccb0640 1 -- 172.21.15.16:0/746730391 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f0d84065430 msgr2=0x7f0d840678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:02:22.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:22.266+0000 7f0d9ccb0640 1 --2- 172.21.15.16:0/746730391 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f0d84065430 0x7f0d840678e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f0d9810ec30 tx=0x7f0d80035040 comp rx=0 tx=0).stop 2024-06-18T03:02:22.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:22.266+0000 7f0d9ccb0640 1 -- 172.21.15.16:0/746730391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d981022c0 msgr2=0x7f0d98113370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:02:22.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:22.266+0000 7f0d9ccb0640 1 --2- 172.21.15.16:0/746730391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d981022c0 0x7f0d98113370 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f0d88002a50 tx=0x7f0d88004600 comp rx=0 tx=0).stop 2024-06-18T03:02:22.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:22.266+0000 7f0d9ccb0640 1 -- 172.21.15.16:0/746730391 shutdown_connections 2024-06-18T03:02:22.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:22.266+0000 7f0d9ccb0640 1 --2- 172.21.15.16:0/746730391 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f0d84065430 0x7f0d840678e0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:02:22.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:22.266+0000 7f0d9ccb0640 1 --2- 172.21.15.16:0/746730391 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d98102c60 0x7f0d981138b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:02:22.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:22.266+0000 7f0d9ccb0640 1 --2- 172.21.15.16:0/746730391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d981022c0 0x7f0d98113370 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:02:22.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:22.266+0000 7f0d9ccb0640 1 -- 172.21.15.16:0/746730391 >> 172.21.15.16:0/746730391 conn(0x7f0d980f7e50 msgr2=0x7f0d980f8db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:02:22.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:22.267+0000 7f0d9ccb0640 1 -- 172.21.15.16:0/746730391 shutdown_connections 2024-06-18T03:02:22.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:22.267+0000 7f0d9ccb0640 1 -- 172.21.15.16:0/746730391 wait complete. 2024-06-18T03:02:22.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:02:22 smithi016 ceph-mon[29214]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:02:22.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:02:22 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/746730391' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:02:22.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:02:22 smithi123 ceph-mon[36358]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:02:22.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:02:22 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/746730391' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:02:24.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:02:24 smithi016 ceph-mon[29214]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:02:24.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:02:24 smithi123 ceph-mon[36358]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:02:25.871 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T03:02:26.150 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T03:02:26.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:02:26 smithi016 ceph-mon[29214]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:02:26.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:02:26 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T03:02:26.699 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:02:26 smithi123 ceph-mon[36358]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:02:26.699 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:02:26 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T03:02:27.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:27.673+0000 7fcb5e85f640 1 -- 172.21.15.16:0/1201665909 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb58102c60 msgr2=0x7fcb580fc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:02:27.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:27.673+0000 7fcb5e85f640 1 --2- 172.21.15.16:0/1201665909 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb58102c60 0x7fcb580fc2f0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c0048d0 tx=0x7fcb4c009da0 comp rx=0 tx=0).stop 2024-06-18T03:02:27.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:27.674+0000 7fcb5e85f640 1 -- 172.21.15.16:0/1201665909 shutdown_connections 2024-06-18T03:02:27.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:27.674+0000 7fcb5e85f640 1 --2- 172.21.15.16:0/1201665909 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb58102c60 0x7fcb580fc2f0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:02:27.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:27.674+0000 7fcb5e85f640 1 --2- 172.21.15.16:0/1201665909 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcb581022c0 0x7fcb58102690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:02:27.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:27.674+0000 7fcb5e85f640 1 -- 172.21.15.16:0/1201665909 >> 172.21.15.16:0/1201665909 conn(0x7fcb580f7e50 msgr2=0x7fcb580fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:02:27.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:27.674+0000 7fcb5e85f640 1 -- 172.21.15.16:0/1201665909 shutdown_connections 2024-06-18T03:02:27.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:27.674+0000 7fcb5e85f640 1 -- 172.21.15.16:0/1201665909 wait complete. 2024-06-18T03:02:27.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:27.674+0000 7fcb5e85f640 1 Processor -- start 2024-06-18T03:02:27.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:27.674+0000 7fcb5e85f640 1 -- start start 2024-06-18T03:02:27.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:27.675+0000 7fcb5e85f640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcb581022c0 0x7fcb5806f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:02:27.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:27.675+0000 7fcb5e85f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb58102c60 0x7fcb5806fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:02:27.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:27.675+0000 7fcb5e85f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb580701f0 con 0x7fcb58102c60 2024-06-18T03:02:27.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:27.675+0000 7fcb5e85f640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb58070360 con 0x7fcb581022c0 2024-06-18T03:02:27.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:27.675+0000 7fcb577fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb58102c60 0x7fcb5806fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:02:27.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:27.675+0000 7fcb577fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb58102c60 0x7fcb5806fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58652/0 (socket says 172.21.15.16:58652) 2024-06-18T03:02:27.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:27.675+0000 7fcb577fe640 1 -- 172.21.15.16:0/3928868231 learned_addr learned my addr 172.21.15.16:0/3928868231 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T03:02:27.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:27.675+0000 7fcb57fff640 1 --2- 172.21.15.16:0/3928868231 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcb581022c0 0x7fcb5806f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:02:27.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:27.675+0000 7fcb577fe640 1 -- 172.21.15.16:0/3928868231 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcb581022c0 msgr2=0x7fcb5806f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:02:27.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:27.675+0000 7fcb577fe640 1 --2- 172.21.15.16:0/3928868231 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcb581022c0 0x7fcb5806f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:02:27.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:27.675+0000 7fcb577fe640 1 -- 172.21.15.16:0/3928868231 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb4c005040 con 0x7fcb58102c60 2024-06-18T03:02:27.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:27.675+0000 7fcb577fe640 1 --2- 172.21.15.16:0/3928868231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb58102c60 0x7fcb5806fcb0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c0043d0 tx=0x7fcb4c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:02:27.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:27.675+0000 7fcb54ff9640 1 -- 172.21.15.16:0/3928868231 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcb4c038070 con 0x7fcb58102c60 2024-06-18T03:02:27.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:27.675+0000 7fcb5e85f640 1 -- 172.21.15.16:0/3928868231 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb58070590 con 0x7fcb58102c60 2024-06-18T03:02:27.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:27.675+0000 7fcb54ff9640 1 -- 172.21.15.16:0/3928868231 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcb4c032970 con 0x7fcb58102c60 2024-06-18T03:02:27.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:27.675+0000 7fcb54ff9640 1 -- 172.21.15.16:0/3928868231 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcb4c03da60 con 0x7fcb58102c60 2024-06-18T03:02:27.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:27.675+0000 7fcb5e85f640 1 -- 172.21.15.16:0/3928868231 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb581199d0 con 0x7fcb58102c60 2024-06-18T03:02:27.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:27.676+0000 7fcb5e85f640 1 -- 172.21.15.16:0/3928868231 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb580fda30 con 0x7fcb58102c60 2024-06-18T03:02:27.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:27.677+0000 7fcb54ff9640 1 -- 172.21.15.16:0/3928868231 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fcb4c030020 con 0x7fcb58102c60 2024-06-18T03:02:27.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:27.677+0000 7fcb54ff9640 1 --2- 172.21.15.16:0/3928868231 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fcb40065430 0x7fcb400678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:02:27.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:27.677+0000 7fcb54ff9640 1 -- 172.21.15.16:0/3928868231 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fcb4c0a26c0 con 0x7fcb58102c60 2024-06-18T03:02:27.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:27.677+0000 7fcb57fff640 1 --2- 172.21.15.16:0/3928868231 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fcb40065430 0x7fcb400678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:02:27.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:27.678+0000 7fcb57fff640 1 --2- 172.21.15.16:0/3928868231 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fcb40065430 0x7fcb400678e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fcb48007fd0 tx=0x7fcb48009e90 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:02:27.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:27.682+0000 7fcb54ff9640 1 -- 172.21.15.16:0/3928868231 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fcb4c070b70 con 0x7fcb58102c60 2024-06-18T03:02:27.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:27.890+0000 7fcb5e85f640 1 -- 172.21.15.16:0/3928868231 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fcb580fbab0 con 0x7fcb58102c60 2024-06-18T03:02:27.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:27.891+0000 7fcb54ff9640 1 -- 172.21.15.16:0/3928868231 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fcb4c074190 con 0x7fcb58102c60 2024-06-18T03:02:27.891 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T03:02:27.891 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-18T03:02:27.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:27.893+0000 7fcb5e85f640 1 -- 172.21.15.16:0/3928868231 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fcb40065430 msgr2=0x7fcb400678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:02:27.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:27.893+0000 7fcb5e85f640 1 --2- 172.21.15.16:0/3928868231 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fcb40065430 0x7fcb400678e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fcb48007fd0 tx=0x7fcb48009e90 comp rx=0 tx=0).stop 2024-06-18T03:02:27.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:27.893+0000 7fcb5e85f640 1 -- 172.21.15.16:0/3928868231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb58102c60 msgr2=0x7fcb5806fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:02:27.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:27.893+0000 7fcb5e85f640 1 --2- 172.21.15.16:0/3928868231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb58102c60 0x7fcb5806fcb0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c0043d0 tx=0x7fcb4c009da0 comp rx=0 tx=0).stop 2024-06-18T03:02:27.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:27.893+0000 7fcb5e85f640 1 -- 172.21.15.16:0/3928868231 shutdown_connections 2024-06-18T03:02:27.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:27.893+0000 7fcb5e85f640 1 --2- 172.21.15.16:0/3928868231 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fcb40065430 0x7fcb400678e0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:02:27.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:27.893+0000 7fcb5e85f640 1 --2- 172.21.15.16:0/3928868231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb58102c60 0x7fcb5806fcb0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:02:27.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:27.893+0000 7fcb5e85f640 1 --2- 172.21.15.16:0/3928868231 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcb581022c0 0x7fcb5806f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:02:27.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:27.893+0000 7fcb5e85f640 1 -- 172.21.15.16:0/3928868231 >> 172.21.15.16:0/3928868231 conn(0x7fcb580f7e50 msgr2=0x7fcb580f8e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:02:27.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:27.893+0000 7fcb5e85f640 1 -- 172.21.15.16:0/3928868231 shutdown_connections 2024-06-18T03:02:27.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:27.893+0000 7fcb5e85f640 1 -- 172.21.15.16:0/3928868231 wait complete. 2024-06-18T03:02:28.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:02:28 smithi016 ceph-mon[29214]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:02:28.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:02:28 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/3928868231' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:02:28.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:02:28 smithi123 ceph-mon[36358]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:02:28.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:02:28 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/3928868231' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:02:30.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:02:30 smithi123 ceph-mon[36358]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:02:30.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:02:30 smithi016 ceph-mon[29214]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:02:31.517 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T03:02:31.795 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T03:02:32.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:02:32 smithi016 ceph-mon[29214]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:02:32.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:02:32 smithi123 ceph-mon[36358]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:02:33.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:33.496+0000 7ff7e1fa0640 1 -- 172.21.15.16:0/859035067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7dc0faa50 msgr2=0x7ff7dc0faec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:02:33.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:33.496+0000 7ff7e1fa0640 1 --2- 172.21.15.16:0/859035067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7dc0faa50 0x7ff7dc0faec0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7ff7d00048d0 tx=0x7ff7d0009da0 comp rx=0 tx=0).stop 2024-06-18T03:02:33.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:33.496+0000 7ff7e1fa0640 1 -- 172.21.15.16:0/859035067 shutdown_connections 2024-06-18T03:02:33.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:33.496+0000 7ff7e1fa0640 1 --2- 172.21.15.16:0/859035067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7dc0faa50 0x7ff7dc0faec0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:02:33.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:33.496+0000 7ff7e1fa0640 1 --2- 172.21.15.16:0/859035067 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff7dc0fd170 0x7ff7dc0fd540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:02:33.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:33.496+0000 7ff7e1fa0640 1 -- 172.21.15.16:0/859035067 >> 172.21.15.16:0/859035067 conn(0x7ff7dc070ab0 msgr2=0x7ff7dc070eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:02:33.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:33.496+0000 7ff7e1fa0640 1 -- 172.21.15.16:0/859035067 shutdown_connections 2024-06-18T03:02:33.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:33.496+0000 7ff7e1fa0640 1 -- 172.21.15.16:0/859035067 wait complete. 2024-06-18T03:02:33.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:33.496+0000 7ff7e1fa0640 1 Processor -- start 2024-06-18T03:02:33.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:33.497+0000 7ff7e1fa0640 1 -- start start 2024-06-18T03:02:33.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:33.497+0000 7ff7e1fa0640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff7dc0faa50 0x7ff7dc1152b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:02:33.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:33.497+0000 7ff7e1fa0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7dc0fd170 0x7ff7dc1157f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:02:33.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:33.497+0000 7ff7e1fa0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7dc115e80 con 0x7ff7dc0fd170 2024-06-18T03:02:33.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:33.497+0000 7ff7e1fa0640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7dc10f230 con 0x7ff7dc0faa50 2024-06-18T03:02:33.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:33.497+0000 7ff7daffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7dc0fd170 0x7ff7dc1157f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:02:33.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:33.497+0000 7ff7daffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7dc0fd170 0x7ff7dc1157f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:32986/0 (socket says 172.21.15.16:32986) 2024-06-18T03:02:33.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:33.497+0000 7ff7daffd640 1 -- 172.21.15.16:0/1202147627 learned_addr learned my addr 172.21.15.16:0/1202147627 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T03:02:33.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:33.497+0000 7ff7daffd640 1 -- 172.21.15.16:0/1202147627 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff7dc0faa50 msgr2=0x7ff7dc1152b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:02:33.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:33.497+0000 7ff7db7fe640 1 --2- 172.21.15.16:0/1202147627 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff7dc0faa50 0x7ff7dc1152b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:02:33.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:33.497+0000 7ff7daffd640 1 --2- 172.21.15.16:0/1202147627 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff7dc0faa50 0x7ff7dc1152b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:02:33.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:33.497+0000 7ff7daffd640 1 -- 172.21.15.16:0/1202147627 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7d0005040 con 0x7ff7dc0fd170 2024-06-18T03:02:33.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:33.498+0000 7ff7daffd640 1 --2- 172.21.15.16:0/1202147627 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7dc0fd170 0x7ff7dc1157f0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7ff7d0003080 tx=0x7ff7d0009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:02:33.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:33.498+0000 7ff7c3fff640 1 -- 172.21.15.16:0/1202147627 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff7d0038070 con 0x7ff7dc0fd170 2024-06-18T03:02:33.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:33.498+0000 7ff7c3fff640 1 -- 172.21.15.16:0/1202147627 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff7d0008ee0 con 0x7ff7dc0fd170 2024-06-18T03:02:33.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:33.498+0000 7ff7e1fa0640 1 -- 172.21.15.16:0/1202147627 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7dc10f4b0 con 0x7ff7dc0fd170 2024-06-18T03:02:33.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:33.498+0000 7ff7c3fff640 1 -- 172.21.15.16:0/1202147627 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff7d003d870 con 0x7ff7dc0fd170 2024-06-18T03:02:33.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:33.498+0000 7ff7e1fa0640 1 -- 172.21.15.16:0/1202147627 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7dc10f970 con 0x7ff7dc0fd170 2024-06-18T03:02:33.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:33.499+0000 7ff7e1fa0640 1 -- 172.21.15.16:0/1202147627 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7dc0fba60 con 0x7ff7dc0fd170 2024-06-18T03:02:33.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:33.499+0000 7ff7c3fff640 1 -- 172.21.15.16:0/1202147627 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7ff7d0030020 con 0x7ff7dc0fd170 2024-06-18T03:02:33.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:33.500+0000 7ff7c3fff640 1 --2- 172.21.15.16:0/1202147627 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7ff7c4065430 0x7ff7c40678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:02:33.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:33.500+0000 7ff7c3fff640 1 -- 172.21.15.16:0/1202147627 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7ff7d00a21b0 con 0x7ff7dc0fd170 2024-06-18T03:02:33.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:33.500+0000 7ff7db7fe640 1 --2- 172.21.15.16:0/1202147627 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7ff7c4065430 0x7ff7c40678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:02:33.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:33.500+0000 7ff7db7fe640 1 --2- 172.21.15.16:0/1202147627 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7ff7c4065430 0x7ff7c40678e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff7cc006fd0 tx=0x7ff7cc008040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:02:33.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:33.504+0000 7ff7c3fff640 1 -- 172.21.15.16:0/1202147627 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7ff7d00706e0 con 0x7ff7dc0fd170 2024-06-18T03:02:33.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:33.712+0000 7ff7e1fa0640 1 -- 172.21.15.16:0/1202147627 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff7dc0749d0 con 0x7ff7dc0fd170 2024-06-18T03:02:33.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:33.713+0000 7ff7c3fff640 1 -- 172.21.15.16:0/1202147627 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7ff7d0073d00 con 0x7ff7dc0fd170 2024-06-18T03:02:33.713 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T03:02:33.713 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-18T03:02:33.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:33.714+0000 7ff7e1fa0640 1 -- 172.21.15.16:0/1202147627 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7ff7c4065430 msgr2=0x7ff7c40678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:02:33.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:33.714+0000 7ff7e1fa0640 1 --2- 172.21.15.16:0/1202147627 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7ff7c4065430 0x7ff7c40678e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff7cc006fd0 tx=0x7ff7cc008040 comp rx=0 tx=0).stop 2024-06-18T03:02:33.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:33.714+0000 7ff7e1fa0640 1 -- 172.21.15.16:0/1202147627 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7dc0fd170 msgr2=0x7ff7dc1157f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:02:33.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:33.714+0000 7ff7e1fa0640 1 --2- 172.21.15.16:0/1202147627 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7dc0fd170 0x7ff7dc1157f0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7ff7d0003080 tx=0x7ff7d0009da0 comp rx=0 tx=0).stop 2024-06-18T03:02:33.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:33.714+0000 7ff7e1fa0640 1 -- 172.21.15.16:0/1202147627 shutdown_connections 2024-06-18T03:02:33.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:33.714+0000 7ff7e1fa0640 1 --2- 172.21.15.16:0/1202147627 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7ff7c4065430 0x7ff7c40678e0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:02:33.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:33.714+0000 7ff7e1fa0640 1 --2- 172.21.15.16:0/1202147627 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff7dc0fd170 0x7ff7dc1157f0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:02:33.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:33.714+0000 7ff7e1fa0640 1 --2- 172.21.15.16:0/1202147627 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff7dc0faa50 0x7ff7dc1152b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:02:33.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:33.714+0000 7ff7e1fa0640 1 -- 172.21.15.16:0/1202147627 >> 172.21.15.16:0/1202147627 conn(0x7ff7dc070ab0 msgr2=0x7ff7dc0f8ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:02:33.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:33.715+0000 7ff7e1fa0640 1 -- 172.21.15.16:0/1202147627 shutdown_connections 2024-06-18T03:02:33.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:33.715+0000 7ff7e1fa0640 1 -- 172.21.15.16:0/1202147627 wait complete. 2024-06-18T03:02:34.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:02:34 smithi016 ceph-mon[29214]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:02:34.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:02:34 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/1202147627' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:02:34.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:02:34 smithi123 ceph-mon[36358]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:02:34.735 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:02:34 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/1202147627' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:02:36.699 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:02:36 smithi123 ceph-mon[36358]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:02:36.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:02:36 smithi016 ceph-mon[29214]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:02:37.450 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T03:02:37.730 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T03:02:38.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:02:38 smithi123 ceph-mon[36358]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:02:38.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:02:38 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T03:02:38.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:02:38 smithi016 ceph-mon[29214]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:02:38.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:02:38 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-18T03:02:39.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:39.427+0000 7f1ad4964640 1 -- 172.21.15.16:0/4256164793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ad01022f0 msgr2=0x7f1ad01026c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:02:39.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:39.427+0000 7f1ad4964640 1 --2- 172.21.15.16:0/4256164793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ad01022f0 0x7f1ad01026c0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f1ac00048d0 tx=0x7f1ac0009da0 comp rx=0 tx=0).stop 2024-06-18T03:02:39.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:39.427+0000 7f1ad4964640 1 -- 172.21.15.16:0/4256164793 shutdown_connections 2024-06-18T03:02:39.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:39.427+0000 7f1ad4964640 1 --2- 172.21.15.16:0/4256164793 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ad0102c90 0x7f1ad00fc320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:02:39.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:39.427+0000 7f1ad4964640 1 --2- 172.21.15.16:0/4256164793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ad01022f0 0x7f1ad01026c0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:02:39.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:39.427+0000 7f1ad4964640 1 -- 172.21.15.16:0/4256164793 >> 172.21.15.16:0/4256164793 conn(0x7f1ad00f7f00 msgr2=0x7f1ad00fa310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:02:39.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:39.427+0000 7f1ad4964640 1 -- 172.21.15.16:0/4256164793 shutdown_connections 2024-06-18T03:02:39.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:39.427+0000 7f1ad4964640 1 -- 172.21.15.16:0/4256164793 wait complete. 2024-06-18T03:02:39.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:39.428+0000 7f1ad4964640 1 Processor -- start 2024-06-18T03:02:39.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:39.428+0000 7f1ad4964640 1 -- start start 2024-06-18T03:02:39.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:39.428+0000 7f1ad4964640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ad01022f0 0x7f1ad0112fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:02:39.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:39.428+0000 7f1ad4964640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ad0102c90 0x7f1ad0113500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:02:39.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:39.428+0000 7f1ad4964640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad0113b00 con 0x7f1ad01022f0 2024-06-18T03:02:39.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:39.428+0000 7f1ad4964640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad010cf40 con 0x7f1ad0102c90 2024-06-18T03:02:39.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:39.428+0000 7f1ace59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ad01022f0 0x7f1ad0112fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:02:39.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:39.428+0000 7f1ace59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ad01022f0 0x7f1ad0112fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33014/0 (socket says 172.21.15.16:33014) 2024-06-18T03:02:39.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:39.428+0000 7f1ace59c640 1 -- 172.21.15.16:0/850064369 learned_addr learned my addr 172.21.15.16:0/850064369 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T03:02:39.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:39.428+0000 7f1acdd9b640 1 --2- 172.21.15.16:0/850064369 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ad0102c90 0x7f1ad0113500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:02:39.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:39.428+0000 7f1ace59c640 1 -- 172.21.15.16:0/850064369 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ad0102c90 msgr2=0x7f1ad0113500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:02:39.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:39.428+0000 7f1ace59c640 1 --2- 172.21.15.16:0/850064369 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ad0102c90 0x7f1ad0113500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:02:39.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:39.428+0000 7f1ace59c640 1 -- 172.21.15.16:0/850064369 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ac0005040 con 0x7f1ad01022f0 2024-06-18T03:02:39.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:39.428+0000 7f1acdd9b640 1 --2- 172.21.15.16:0/850064369 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ad0102c90 0x7f1ad0113500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-18T03:02:39.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:39.429+0000 7f1ace59c640 1 --2- 172.21.15.16:0/850064369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ad01022f0 0x7f1ad0112fc0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f1ac00043d0 tx=0x7f1ac0009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:02:39.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:39.429+0000 7f1abeffd640 1 -- 172.21.15.16:0/850064369 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ac0038070 con 0x7f1ad01022f0 2024-06-18T03:02:39.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:39.429+0000 7f1ad4964640 1 -- 172.21.15.16:0/850064369 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ad010d1c0 con 0x7f1ad01022f0 2024-06-18T03:02:39.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:39.429+0000 7f1abeffd640 1 -- 172.21.15.16:0/850064369 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1ac0032970 con 0x7f1ad01022f0 2024-06-18T03:02:39.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:39.429+0000 7f1abeffd640 1 -- 172.21.15.16:0/850064369 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ac003da60 con 0x7f1ad01022f0 2024-06-18T03:02:39.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:39.429+0000 7f1ad4964640 1 -- 172.21.15.16:0/850064369 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ad010d700 con 0x7f1ad01022f0 2024-06-18T03:02:39.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:39.430+0000 7f1ad4964640 1 -- 172.21.15.16:0/850064369 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ad00fda60 con 0x7f1ad01022f0 2024-06-18T03:02:39.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:39.430+0000 7f1abeffd640 1 -- 172.21.15.16:0/850064369 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f1ac0030020 con 0x7f1ad01022f0 2024-06-18T03:02:39.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:39.431+0000 7f1abeffd640 1 --2- 172.21.15.16:0/850064369 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f1ab8065430 0x7f1ab80678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:02:39.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:39.431+0000 7f1abeffd640 1 -- 172.21.15.16:0/850064369 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f1ac00a26d0 con 0x7f1ad01022f0 2024-06-18T03:02:39.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:39.431+0000 7f1acdd9b640 1 --2- 172.21.15.16:0/850064369 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f1ab8065430 0x7f1ab80678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:02:39.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:39.431+0000 7f1acdd9b640 1 --2- 172.21.15.16:0/850064369 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f1ab8065430 0x7f1ab80678e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1ac4009fd0 tx=0x7f1ac4009e50 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:02:39.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:39.435+0000 7f1abeffd640 1 -- 172.21.15.16:0/850064369 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f1ac0070b80 con 0x7f1ad01022f0 2024-06-18T03:02:39.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:02:39 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T03:02:39.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:02:39 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T03:02:39.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:02:39 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T03:02:39.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:39.649+0000 7f1ad4964640 1 -- 172.21.15.16:0/850064369 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f1ad004b480 con 0x7f1ad01022f0 2024-06-18T03:02:39.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:39.650+0000 7f1abeffd640 1 -- 172.21.15.16:0/850064369 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f1ac00741a0 con 0x7f1ad01022f0 2024-06-18T03:02:39.650 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T03:02:39.650 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-18T03:02:39.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:39.652+0000 7f1ad4964640 1 -- 172.21.15.16:0/850064369 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f1ab8065430 msgr2=0x7f1ab80678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:02:39.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:39.652+0000 7f1ad4964640 1 --2- 172.21.15.16:0/850064369 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f1ab8065430 0x7f1ab80678e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1ac4009fd0 tx=0x7f1ac4009e50 comp rx=0 tx=0).stop 2024-06-18T03:02:39.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:39.652+0000 7f1ad4964640 1 -- 172.21.15.16:0/850064369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ad01022f0 msgr2=0x7f1ad0112fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:02:39.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:39.652+0000 7f1ad4964640 1 --2- 172.21.15.16:0/850064369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ad01022f0 0x7f1ad0112fc0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f1ac00043d0 tx=0x7f1ac0009da0 comp rx=0 tx=0).stop 2024-06-18T03:02:39.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:39.652+0000 7f1ad4964640 1 -- 172.21.15.16:0/850064369 shutdown_connections 2024-06-18T03:02:39.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:39.652+0000 7f1ad4964640 1 --2- 172.21.15.16:0/850064369 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f1ab8065430 0x7f1ab80678e0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:02:39.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:39.652+0000 7f1ad4964640 1 --2- 172.21.15.16:0/850064369 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1ad0102c90 0x7f1ad0113500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:02:39.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:39.652+0000 7f1ad4964640 1 --2- 172.21.15.16:0/850064369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ad01022f0 0x7f1ad0112fc0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:02:39.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:39.652+0000 7f1ad4964640 1 -- 172.21.15.16:0/850064369 >> 172.21.15.16:0/850064369 conn(0x7f1ad00f7f00 msgr2=0x7f1ad00f8e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:02:39.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:39.652+0000 7f1ad4964640 1 -- 172.21.15.16:0/850064369 shutdown_connections 2024-06-18T03:02:39.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:39.652+0000 7f1ad4964640 1 -- 172.21.15.16:0/850064369 wait complete. 2024-06-18T03:02:39.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:02:39 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-18T03:02:39.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:02:39 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-18T03:02:39.734 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:02:39 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T03:02:40.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:02:40 smithi016 ceph-mon[29214]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:02:40.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:02:40 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/850064369' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:02:40.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:02:40 smithi123 ceph-mon[36358]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:02:40.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:02:40 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/850064369' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:02:41.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:02:41 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T03:02:42.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:02:41 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T03:02:42.957 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:02:42 smithi016 ceph-mon[29214]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:02:42.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:02:42 smithi123 ceph-mon[36358]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:02:43.475 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T03:02:43.755 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T03:02:44.950 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:02:44 smithi016 ceph-mon[29214]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:02:44.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:02:44 smithi123 ceph-mon[36358]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:02:45.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:45.495+0000 7fd70d9a3640 1 -- 172.21.15.16:0/3331098466 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7080fb570 msgr2=0x7fd7080fb940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:02:45.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:45.495+0000 7fd70d9a3640 1 --2- 172.21.15.16:0/3331098466 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7080fb570 0x7fd7080fb940 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fd6f80048d0 tx=0x7fd6f8009da0 comp rx=0 tx=0).stop 2024-06-18T03:02:45.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:45.495+0000 7fd70d9a3640 1 -- 172.21.15.16:0/3331098466 shutdown_connections 2024-06-18T03:02:45.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:45.495+0000 7fd70d9a3640 1 --2- 172.21.15.16:0/3331098466 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7080fbe80 0x7fd7080fcd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:02:45.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:45.495+0000 7fd70d9a3640 1 --2- 172.21.15.16:0/3331098466 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7080fb570 0x7fd7080fb940 secure :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fd6f80048d0 tx=0x7fd6f8009da0 comp rx=0 tx=0).stop 2024-06-18T03:02:45.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:45.495+0000 7fd70d9a3640 1 -- 172.21.15.16:0/3331098466 >> 172.21.15.16:0/3331098466 conn(0x7fd708072110 msgr2=0x7fd708072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:02:45.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:45.495+0000 7fd70d9a3640 1 -- 172.21.15.16:0/3331098466 shutdown_connections 2024-06-18T03:02:45.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:45.495+0000 7fd70d9a3640 1 -- 172.21.15.16:0/3331098466 wait complete. 2024-06-18T03:02:45.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:45.496+0000 7fd70d9a3640 1 Processor -- start 2024-06-18T03:02:45.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:45.496+0000 7fd70d9a3640 1 -- start start 2024-06-18T03:02:45.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:45.496+0000 7fd70d9a3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7080fbe80 0x7fd7081153b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:02:45.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:45.496+0000 7fd70d9a3640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7081158f0 0x7fd70810f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:02:45.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:45.496+0000 7fd70d9a3640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd708115ee0 con 0x7fd7080fbe80 2024-06-18T03:02:45.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:45.496+0000 7fd70d9a3640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd708116050 con 0x7fd7081158f0 2024-06-18T03:02:45.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:45.497+0000 7fd706ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7080fbe80 0x7fd7081153b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:02:45.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:45.497+0000 7fd706ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7080fbe80 0x7fd7081153b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51146/0 (socket says 172.21.15.16:51146) 2024-06-18T03:02:45.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:45.497+0000 7fd706ffd640 1 -- 172.21.15.16:0/186479818 learned_addr learned my addr 172.21.15.16:0/186479818 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T03:02:45.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:45.497+0000 7fd7067fc640 1 --2- 172.21.15.16:0/186479818 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7081158f0 0x7fd70810f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:02:45.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:45.497+0000 7fd706ffd640 1 -- 172.21.15.16:0/186479818 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7081158f0 msgr2=0x7fd70810f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:02:45.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:45.497+0000 7fd706ffd640 1 --2- 172.21.15.16:0/186479818 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7081158f0 0x7fd70810f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:02:45.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:45.497+0000 7fd706ffd640 1 -- 172.21.15.16:0/186479818 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6f8005040 con 0x7fd7080fbe80 2024-06-18T03:02:45.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:45.497+0000 7fd706ffd640 1 --2- 172.21.15.16:0/186479818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7080fbe80 0x7fd7081153b0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fd6f8003080 tx=0x7fd6f8009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:02:45.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:45.497+0000 7fd6ef7fe640 1 -- 172.21.15.16:0/186479818 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd6f8038070 con 0x7fd7080fbe80 2024-06-18T03:02:45.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:45.497+0000 7fd6ef7fe640 1 -- 172.21.15.16:0/186479818 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd6f8009890 con 0x7fd7080fbe80 2024-06-18T03:02:45.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:45.497+0000 7fd70d9a3640 1 -- 172.21.15.16:0/186479818 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd70810f910 con 0x7fd7080fbe80 2024-06-18T03:02:45.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:45.497+0000 7fd6ef7fe640 1 -- 172.21.15.16:0/186479818 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd6f803d7b0 con 0x7fd7080fbe80 2024-06-18T03:02:45.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:45.497+0000 7fd70d9a3640 1 -- 172.21.15.16:0/186479818 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd70810fe30 con 0x7fd7080fbe80 2024-06-18T03:02:45.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:45.498+0000 7fd70d9a3640 1 -- 172.21.15.16:0/186479818 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7080fc920 con 0x7fd7080fbe80 2024-06-18T03:02:45.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:45.499+0000 7fd6ef7fe640 1 -- 172.21.15.16:0/186479818 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fd6f8030020 con 0x7fd7080fbe80 2024-06-18T03:02:45.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:45.499+0000 7fd6ef7fe640 1 --2- 172.21.15.16:0/186479818 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fd6f0065430 0x7fd6f00678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:02:45.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:45.499+0000 7fd6ef7fe640 1 -- 172.21.15.16:0/186479818 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fd6f80a20e0 con 0x7fd7080fbe80 2024-06-18T03:02:45.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:45.499+0000 7fd7067fc640 1 --2- 172.21.15.16:0/186479818 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fd6f0065430 0x7fd6f00678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:02:45.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:45.500+0000 7fd7067fc640 1 --2- 172.21.15.16:0/186479818 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fd6f0065430 0x7fd6f00678e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd6fc006fd0 tx=0x7fd6fc008040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:02:45.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:45.504+0000 7fd6ef7fe640 1 -- 172.21.15.16:0/186479818 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fd6f80706b0 con 0x7fd7080fbe80 2024-06-18T03:02:45.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:45.713+0000 7fd70d9a3640 1 -- 172.21.15.16:0/186479818 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd708110a60 con 0x7fd7080fbe80 2024-06-18T03:02:45.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:45.713+0000 7fd6ef7fe640 1 -- 172.21.15.16:0/186479818 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fd6f8073cd0 con 0x7fd7080fbe80 2024-06-18T03:02:45.714 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T03:02:45.714 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-18T03:02:45.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:45.715+0000 7fd70d9a3640 1 -- 172.21.15.16:0/186479818 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fd6f0065430 msgr2=0x7fd6f00678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:02:45.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:45.715+0000 7fd70d9a3640 1 --2- 172.21.15.16:0/186479818 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fd6f0065430 0x7fd6f00678e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd6fc006fd0 tx=0x7fd6fc008040 comp rx=0 tx=0).stop 2024-06-18T03:02:45.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:45.715+0000 7fd70d9a3640 1 -- 172.21.15.16:0/186479818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7080fbe80 msgr2=0x7fd7081153b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:02:45.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:45.715+0000 7fd70d9a3640 1 --2- 172.21.15.16:0/186479818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7080fbe80 0x7fd7081153b0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fd6f8003080 tx=0x7fd6f8009da0 comp rx=0 tx=0).stop 2024-06-18T03:02:45.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:45.715+0000 7fd70d9a3640 1 -- 172.21.15.16:0/186479818 shutdown_connections 2024-06-18T03:02:45.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:45.715+0000 7fd70d9a3640 1 --2- 172.21.15.16:0/186479818 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fd6f0065430 0x7fd6f00678e0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:02:45.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:45.715+0000 7fd70d9a3640 1 --2- 172.21.15.16:0/186479818 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7081158f0 0x7fd70810f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:02:45.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:45.715+0000 7fd70d9a3640 1 --2- 172.21.15.16:0/186479818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7080fbe80 0x7fd7081153b0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:02:45.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:45.715+0000 7fd70d9a3640 1 -- 172.21.15.16:0/186479818 >> 172.21.15.16:0/186479818 conn(0x7fd708072110 msgr2=0x7fd7080fde90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:02:45.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:45.715+0000 7fd70d9a3640 1 -- 172.21.15.16:0/186479818 shutdown_connections 2024-06-18T03:02:45.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:45.715+0000 7fd70d9a3640 1 -- 172.21.15.16:0/186479818 wait complete. 2024-06-18T03:02:46.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:02:46 smithi016 ceph-mon[29214]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:02:46.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:02:46 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/186479818' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:02:46.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:02:46 smithi123 ceph-mon[36358]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:02:46.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:02:46 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/186479818' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:02:48.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:02:48 smithi123 ceph-mon[36358]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:02:49.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:02:48 smithi016 ceph-mon[29214]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:02:49.490 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T03:02:49.767 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T03:02:50.931 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:02:50 smithi016 ceph-mon[29214]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:02:50.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:02:50 smithi123 ceph-mon[36358]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:02:51.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:51.456+0000 7fcef1ed0640 1 -- 172.21.15.16:0/2875009493 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fceec101b60 msgr2=0x7fceec0fb1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:02:51.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:51.456+0000 7fcef1ed0640 1 --2- 172.21.15.16:0/2875009493 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fceec101b60 0x7fceec0fb1f0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fcee00048d0 tx=0x7fcee0009da0 comp rx=0 tx=0).stop 2024-06-18T03:02:51.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:51.456+0000 7fcef1ed0640 1 -- 172.21.15.16:0/2875009493 shutdown_connections 2024-06-18T03:02:51.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:51.456+0000 7fcef1ed0640 1 --2- 172.21.15.16:0/2875009493 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fceec101b60 0x7fceec0fb1f0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:02:51.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:51.456+0000 7fcef1ed0640 1 --2- 172.21.15.16:0/2875009493 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fceec1011c0 0x7fceec101590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:02:51.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:51.456+0000 7fcef1ed0640 1 -- 172.21.15.16:0/2875009493 >> 172.21.15.16:0/2875009493 conn(0x7fceec0f6db0 msgr2=0x7fceec0f91c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:02:51.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:51.456+0000 7fcef1ed0640 1 -- 172.21.15.16:0/2875009493 shutdown_connections 2024-06-18T03:02:51.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:51.456+0000 7fcef1ed0640 1 -- 172.21.15.16:0/2875009493 wait complete. 2024-06-18T03:02:51.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:51.457+0000 7fcef1ed0640 1 Processor -- start 2024-06-18T03:02:51.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:51.457+0000 7fcef1ed0640 1 -- start start 2024-06-18T03:02:51.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:51.457+0000 7fcef1ed0640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fceec1011c0 0x7fceec106080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:02:51.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:51.457+0000 7fcef1ed0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fceec101b60 0x7fceec1065c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:02:51.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:51.457+0000 7fcef1ed0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceec106b00 con 0x7fceec101b60 2024-06-18T03:02:51.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:51.457+0000 7fcef1ed0640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceec106c70 con 0x7fceec1011c0 2024-06-18T03:02:51.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:51.457+0000 7fceeaffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fceec101b60 0x7fceec1065c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:02:51.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:51.457+0000 7fceeaffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fceec101b60 0x7fceec1065c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35594/0 (socket says 172.21.15.16:35594) 2024-06-18T03:02:51.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:51.457+0000 7fceeaffd640 1 -- 172.21.15.16:0/2014262983 learned_addr learned my addr 172.21.15.16:0/2014262983 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T03:02:51.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:51.457+0000 7fceeb7fe640 1 --2- 172.21.15.16:0/2014262983 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fceec1011c0 0x7fceec106080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:02:51.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:51.457+0000 7fceeaffd640 1 -- 172.21.15.16:0/2014262983 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fceec1011c0 msgr2=0x7fceec106080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:02:51.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:51.457+0000 7fceeaffd640 1 --2- 172.21.15.16:0/2014262983 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fceec1011c0 0x7fceec106080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:02:51.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:51.457+0000 7fceeaffd640 1 -- 172.21.15.16:0/2014262983 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcee0005040 con 0x7fceec101b60 2024-06-18T03:02:51.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:51.458+0000 7fceeaffd640 1 --2- 172.21.15.16:0/2014262983 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fceec101b60 0x7fceec1065c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fcee0003080 tx=0x7fcee0009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:02:51.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:51.458+0000 7fced3fff640 1 -- 172.21.15.16:0/2014262983 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcee0038070 con 0x7fceec101b60 2024-06-18T03:02:51.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:51.458+0000 7fcef1ed0640 1 -- 172.21.15.16:0/2014262983 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fceec106ea0 con 0x7fceec101b60 2024-06-18T03:02:51.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:51.458+0000 7fced3fff640 1 -- 172.21.15.16:0/2014262983 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcee0008ee0 con 0x7fceec101b60 2024-06-18T03:02:51.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:51.458+0000 7fced3fff640 1 -- 172.21.15.16:0/2014262983 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcee003d870 con 0x7fceec101b60 2024-06-18T03:02:51.462 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:51.458+0000 7fcef1ed0640 1 -- 172.21.15.16:0/2014262983 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fceec1196b0 con 0x7fceec101b60 2024-06-18T03:02:51.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:51.459+0000 7fcef1ed0640 1 -- 172.21.15.16:0/2014262983 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fceec0fc930 con 0x7fceec101b60 2024-06-18T03:02:51.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:51.459+0000 7fced3fff640 1 -- 172.21.15.16:0/2014262983 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fcee0030020 con 0x7fceec101b60 2024-06-18T03:02:51.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:51.460+0000 7fced3fff640 1 --2- 172.21.15.16:0/2014262983 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fced4065430 0x7fced40678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:02:51.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:51.460+0000 7fced3fff640 1 -- 172.21.15.16:0/2014262983 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fcee00a21b0 con 0x7fceec101b60 2024-06-18T03:02:51.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:51.460+0000 7fceeb7fe640 1 --2- 172.21.15.16:0/2014262983 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fced4065430 0x7fced40678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:02:51.463 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:51.460+0000 7fceeb7fe640 1 --2- 172.21.15.16:0/2014262983 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fced4065430 0x7fced40678e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fcedc007fd0 tx=0x7fcedc009040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:02:51.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:51.464+0000 7fced3fff640 1 -- 172.21.15.16:0/2014262983 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fcee00706e0 con 0x7fceec101b60 2024-06-18T03:02:51.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:51.675+0000 7fcef1ed0640 1 -- 172.21.15.16:0/2014262983 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fceec107770 con 0x7fceec101b60 2024-06-18T03:02:51.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:51.676+0000 7fced3fff640 1 -- 172.21.15.16:0/2014262983 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fcee0073d00 con 0x7fceec101b60 2024-06-18T03:02:51.677 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T03:02:51.678 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-18T03:02:51.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:51.678+0000 7fcef1ed0640 1 -- 172.21.15.16:0/2014262983 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fced4065430 msgr2=0x7fced40678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:02:51.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:51.678+0000 7fcef1ed0640 1 --2- 172.21.15.16:0/2014262983 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fced4065430 0x7fced40678e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fcedc007fd0 tx=0x7fcedc009040 comp rx=0 tx=0).stop 2024-06-18T03:02:51.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:51.678+0000 7fcef1ed0640 1 -- 172.21.15.16:0/2014262983 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fceec101b60 msgr2=0x7fceec1065c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:02:51.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:51.678+0000 7fcef1ed0640 1 --2- 172.21.15.16:0/2014262983 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fceec101b60 0x7fceec1065c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fcee0003080 tx=0x7fcee0009da0 comp rx=0 tx=0).stop 2024-06-18T03:02:51.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:51.678+0000 7fcef1ed0640 1 -- 172.21.15.16:0/2014262983 shutdown_connections 2024-06-18T03:02:51.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:51.678+0000 7fcef1ed0640 1 --2- 172.21.15.16:0/2014262983 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fced4065430 0x7fced40678e0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:02:51.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:51.678+0000 7fcef1ed0640 1 --2- 172.21.15.16:0/2014262983 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fceec101b60 0x7fceec1065c0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:02:51.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:51.678+0000 7fcef1ed0640 1 --2- 172.21.15.16:0/2014262983 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fceec1011c0 0x7fceec106080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:02:51.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:51.678+0000 7fcef1ed0640 1 -- 172.21.15.16:0/2014262983 >> 172.21.15.16:0/2014262983 conn(0x7fceec0f6db0 msgr2=0x7fceec0f7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:02:51.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:51.678+0000 7fcef1ed0640 1 -- 172.21.15.16:0/2014262983 shutdown_connections 2024-06-18T03:02:51.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:51.678+0000 7fcef1ed0640 1 -- 172.21.15.16:0/2014262983 wait complete. 2024-06-18T03:02:52.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:02:52 smithi016 ceph-mon[29214]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:02:52.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:02:52 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/2014262983' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:02:52.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:02:52 smithi123 ceph-mon[36358]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:02:52.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:02:52 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/2014262983' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:02:54.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:02:54 smithi123 ceph-mon[36358]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:02:55.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:02:54 smithi016 ceph-mon[29214]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:02:55.408 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T03:02:55.685 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T03:02:56.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:02:56 smithi123 ceph-mon[36358]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:02:56.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:02:56 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T03:02:56.988 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:02:56 smithi016 ceph-mon[29214]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:02:56.988 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:02:56 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T03:02:57.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:57.303+0000 7f25fbb76640 1 -- 172.21.15.16:0/3089209385 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25f40700c0 msgr2=0x7f25f4108630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:02:57.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:57.303+0000 7f25fbb76640 1 --2- 172.21.15.16:0/3089209385 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25f40700c0 0x7f25f4108630 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f25e40048d0 tx=0x7f25e4009da0 comp rx=0 tx=0).stop 2024-06-18T03:02:57.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:57.303+0000 7f25fbb76640 1 -- 172.21.15.16:0/3089209385 shutdown_connections 2024-06-18T03:02:57.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:57.303+0000 7f25fbb76640 1 --2- 172.21.15.16:0/3089209385 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25f40700c0 0x7f25f4108630 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:02:57.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:57.303+0000 7f25fbb76640 1 --2- 172.21.15.16:0/3089209385 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f25f406f7b0 0x7f25f406fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:02:57.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:57.303+0000 7f25fbb76640 1 -- 172.21.15.16:0/3089209385 >> 172.21.15.16:0/3089209385 conn(0x7f25f40f7e50 msgr2=0x7f25f40fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:02:57.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:57.303+0000 7f25fbb76640 1 -- 172.21.15.16:0/3089209385 shutdown_connections 2024-06-18T03:02:57.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:57.303+0000 7f25fbb76640 1 -- 172.21.15.16:0/3089209385 wait complete. 2024-06-18T03:02:57.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:57.303+0000 7f25fbb76640 1 Processor -- start 2024-06-18T03:02:57.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:57.303+0000 7f25fbb76640 1 -- start start 2024-06-18T03:02:57.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:57.304+0000 7f25fbb76640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25f406f7b0 0x7f25f41152c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:02:57.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:57.304+0000 7f25fbb76640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f25f40700c0 0x7f25f4115800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:02:57.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:57.304+0000 7f25fbb76640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25f4115e90 con 0x7f25f406f7b0 2024-06-18T03:02:57.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:57.304+0000 7f25fbb76640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25f410f240 con 0x7f25f40700c0 2024-06-18T03:02:57.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:57.304+0000 7f25f9912640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25f406f7b0 0x7f25f41152c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:02:57.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:57.304+0000 7f25f9912640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25f406f7b0 0x7f25f41152c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35608/0 (socket says 172.21.15.16:35608) 2024-06-18T03:02:57.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:57.304+0000 7f25f9912640 1 -- 172.21.15.16:0/792822843 learned_addr learned my addr 172.21.15.16:0/792822843 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T03:02:57.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:57.304+0000 7f25f9111640 1 --2- 172.21.15.16:0/792822843 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f25f40700c0 0x7f25f4115800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:02:57.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:57.304+0000 7f25f9912640 1 -- 172.21.15.16:0/792822843 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f25f40700c0 msgr2=0x7f25f4115800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:02:57.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:57.304+0000 7f25f9912640 1 --2- 172.21.15.16:0/792822843 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f25f40700c0 0x7f25f4115800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:02:57.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:57.304+0000 7f25f9912640 1 -- 172.21.15.16:0/792822843 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25e4005040 con 0x7f25f406f7b0 2024-06-18T03:02:57.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:57.304+0000 7f25f9912640 1 --2- 172.21.15.16:0/792822843 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25f406f7b0 0x7f25f41152c0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f25f000b790 tx=0x7f25f00091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:02:57.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:57.304+0000 7f25ea7fc640 1 -- 172.21.15.16:0/792822843 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f25f000c700 con 0x7f25f406f7b0 2024-06-18T03:02:57.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:57.304+0000 7f25ea7fc640 1 -- 172.21.15.16:0/792822843 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f25f000cd30 con 0x7f25f406f7b0 2024-06-18T03:02:57.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:57.304+0000 7f25fbb76640 1 -- 172.21.15.16:0/792822843 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25f410f520 con 0x7f25f406f7b0 2024-06-18T03:02:57.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:57.304+0000 7f25ea7fc640 1 -- 172.21.15.16:0/792822843 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f25f0013370 con 0x7f25f406f7b0 2024-06-18T03:02:57.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:57.304+0000 7f25fbb76640 1 -- 172.21.15.16:0/792822843 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25f410fa10 con 0x7f25f406f7b0 2024-06-18T03:02:57.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:57.305+0000 7f25fbb76640 1 -- 172.21.15.16:0/792822843 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25f4070ef0 con 0x7f25f406f7b0 2024-06-18T03:02:57.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:57.307+0000 7f25ea7fc640 1 -- 172.21.15.16:0/792822843 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f25f0012070 con 0x7f25f406f7b0 2024-06-18T03:02:57.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:57.307+0000 7f25ea7fc640 1 --2- 172.21.15.16:0/792822843 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f25e0065430 0x7f25e00678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:02:57.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:57.307+0000 7f25ea7fc640 1 -- 172.21.15.16:0/792822843 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f25f00820b0 con 0x7f25f406f7b0 2024-06-18T03:02:57.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:57.307+0000 7f25f9111640 1 --2- 172.21.15.16:0/792822843 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f25e0065430 0x7f25e00678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:02:57.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:57.308+0000 7f25f9111640 1 --2- 172.21.15.16:0/792822843 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f25e0065430 0x7f25e00678e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f25e4009f80 tx=0x7f25e4035040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:02:57.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:57.311+0000 7f25ea7fc640 1 -- 172.21.15.16:0/792822843 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f25f004ddf0 con 0x7f25f406f7b0 2024-06-18T03:02:57.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:57.522+0000 7f25fbb76640 1 -- 172.21.15.16:0/792822843 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f25f40fba90 con 0x7f25f406f7b0 2024-06-18T03:02:57.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:57.522+0000 7f25ea7fc640 1 -- 172.21.15.16:0/792822843 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f25f0050680 con 0x7f25f406f7b0 2024-06-18T03:02:57.523 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T03:02:57.523 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-18T03:02:57.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:57.524+0000 7f25fbb76640 1 -- 172.21.15.16:0/792822843 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f25e0065430 msgr2=0x7f25e00678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:02:57.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:57.524+0000 7f25fbb76640 1 --2- 172.21.15.16:0/792822843 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f25e0065430 0x7f25e00678e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f25e4009f80 tx=0x7f25e4035040 comp rx=0 tx=0).stop 2024-06-18T03:02:57.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:57.524+0000 7f25fbb76640 1 -- 172.21.15.16:0/792822843 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25f406f7b0 msgr2=0x7f25f41152c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:02:57.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:57.524+0000 7f25fbb76640 1 --2- 172.21.15.16:0/792822843 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25f406f7b0 0x7f25f41152c0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f25f000b790 tx=0x7f25f00091d0 comp rx=0 tx=0).stop 2024-06-18T03:02:57.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:57.524+0000 7f25fbb76640 1 -- 172.21.15.16:0/792822843 shutdown_connections 2024-06-18T03:02:57.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:57.524+0000 7f25fbb76640 1 --2- 172.21.15.16:0/792822843 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f25e0065430 0x7f25e00678e0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:02:57.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:57.524+0000 7f25fbb76640 1 --2- 172.21.15.16:0/792822843 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f25f40700c0 0x7f25f4115800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:02:57.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:57.524+0000 7f25fbb76640 1 --2- 172.21.15.16:0/792822843 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25f406f7b0 0x7f25f41152c0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:02:57.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:57.524+0000 7f25fbb76640 1 -- 172.21.15.16:0/792822843 >> 172.21.15.16:0/792822843 conn(0x7f25f40f7e50 msgr2=0x7f25f40f92b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:02:57.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:57.525+0000 7f25fbb76640 1 -- 172.21.15.16:0/792822843 shutdown_connections 2024-06-18T03:02:57.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:02:57.525+0000 7f25fbb76640 1 -- 172.21.15.16:0/792822843 wait complete. 2024-06-18T03:02:57.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:02:57 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/792822843' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:02:58.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:02:57 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/792822843' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:02:58.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:02:58 smithi123 ceph-mon[36358]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:02:59.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:02:58 smithi016 ceph-mon[29214]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:03:00.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:03:00 smithi123 ceph-mon[36358]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:03:01.057 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:03:00 smithi016 ceph-mon[29214]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:03:01.244 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T03:03:01.520 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T03:03:02.953 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:03:02 smithi016 ceph-mon[29214]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:03:02.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:03:02 smithi123 ceph-mon[36358]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:03:03.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:03.278+0000 7f65dccfa640 1 -- 172.21.15.16:0/3130749601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65d80fd190 msgr2=0x7f65d80fd560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:03:03.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:03.278+0000 7f65dccfa640 1 --2- 172.21.15.16:0/3130749601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65d80fd190 0x7f65d80fd560 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f65c80048d0 tx=0x7f65c8009da0 comp rx=0 tx=0).stop 2024-06-18T03:03:03.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:03.279+0000 7f65dccfa640 1 -- 172.21.15.16:0/3130749601 shutdown_connections 2024-06-18T03:03:03.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:03.279+0000 7f65dccfa640 1 --2- 172.21.15.16:0/3130749601 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65d80fba50 0x7f65d80fbea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:03:03.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:03.279+0000 7f65dccfa640 1 --2- 172.21.15.16:0/3130749601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65d80fd190 0x7f65d80fd560 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:03:03.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:03.279+0000 7f65dccfa640 1 -- 172.21.15.16:0/3130749601 >> 172.21.15.16:0/3130749601 conn(0x7f65d8071860 msgr2=0x7f65d8071c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:03:03.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:03.279+0000 7f65dccfa640 1 -- 172.21.15.16:0/3130749601 shutdown_connections 2024-06-18T03:03:03.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:03.279+0000 7f65dccfa640 1 -- 172.21.15.16:0/3130749601 wait complete. 2024-06-18T03:03:03.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:03.279+0000 7f65dccfa640 1 Processor -- start 2024-06-18T03:03:03.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:03.279+0000 7f65dccfa640 1 -- start start 2024-06-18T03:03:03.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:03.279+0000 7f65dccfa640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65d80fba50 0x7f65d8113050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:03:03.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:03.279+0000 7f65dccfa640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65d80fd190 0x7f65d8113590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:03:03.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:03.279+0000 7f65dccfa640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65d8113b90 con 0x7f65d80fd190 2024-06-18T03:03:03.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:03.279+0000 7f65dccfa640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65d810cfd0 con 0x7f65d80fba50 2024-06-18T03:03:03.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:03.280+0000 7f65cffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65d80fd190 0x7f65d8113590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:03:03.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:03.280+0000 7f65cffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65d80fd190 0x7f65d8113590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36586/0 (socket says 172.21.15.16:36586) 2024-06-18T03:03:03.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:03.280+0000 7f65cffff640 1 -- 172.21.15.16:0/3621386435 learned_addr learned my addr 172.21.15.16:0/3621386435 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T03:03:03.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:03.280+0000 7f65d659c640 1 --2- 172.21.15.16:0/3621386435 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65d80fba50 0x7f65d8113050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:03:03.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:03.280+0000 7f65cffff640 1 -- 172.21.15.16:0/3621386435 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65d80fba50 msgr2=0x7f65d8113050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:03:03.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:03.280+0000 7f65cffff640 1 --2- 172.21.15.16:0/3621386435 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65d80fba50 0x7f65d8113050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:03:03.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:03.280+0000 7f65cffff640 1 -- 172.21.15.16:0/3621386435 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65c8005040 con 0x7f65d80fd190 2024-06-18T03:03:03.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:03.280+0000 7f65cffff640 1 --2- 172.21.15.16:0/3621386435 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65d80fd190 0x7f65d8113590 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f65c000c790 tx=0x7f65c00043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:03:03.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:03.280+0000 7f65ceffd640 1 -- 172.21.15.16:0/3621386435 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f65c000a7a0 con 0x7f65d80fd190 2024-06-18T03:03:03.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:03.280+0000 7f65dccfa640 1 -- 172.21.15.16:0/3621386435 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65d810d2b0 con 0x7f65d80fd190 2024-06-18T03:03:03.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:03.280+0000 7f65ceffd640 1 -- 172.21.15.16:0/3621386435 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f65c000add0 con 0x7f65d80fd190 2024-06-18T03:03:03.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:03.280+0000 7f65ceffd640 1 -- 172.21.15.16:0/3621386435 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f65c00115e0 con 0x7f65d80fd190 2024-06-18T03:03:03.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:03.280+0000 7f65dccfa640 1 -- 172.21.15.16:0/3621386435 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65d810d7d0 con 0x7f65d80fd190 2024-06-18T03:03:03.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:03.281+0000 7f65dccfa640 1 -- 172.21.15.16:0/3621386435 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65d80fca60 con 0x7f65d80fd190 2024-06-18T03:03:03.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:03.282+0000 7f65ceffd640 1 -- 172.21.15.16:0/3621386435 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f65c0012400 con 0x7f65d80fd190 2024-06-18T03:03:03.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:03.282+0000 7f65ceffd640 1 --2- 172.21.15.16:0/3621386435 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f65c4065430 0x7f65c40678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:03:03.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:03.282+0000 7f65ceffd640 1 -- 172.21.15.16:0/3621386435 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f65c0082560 con 0x7f65d80fd190 2024-06-18T03:03:03.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:03.282+0000 7f65d659c640 1 --2- 172.21.15.16:0/3621386435 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f65c4065430 0x7f65c40678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:03:03.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:03.283+0000 7f65d659c640 1 --2- 172.21.15.16:0/3621386435 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f65c4065430 0x7f65c40678e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f65c8009f80 tx=0x7f65c8035040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:03:03.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:03.286+0000 7f65ceffd640 1 -- 172.21.15.16:0/3621386435 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f65c0050b50 con 0x7f65d80fd190 2024-06-18T03:03:03.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:03.501+0000 7f65dccfa640 1 -- 172.21.15.16:0/3621386435 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f65d810e6c0 con 0x7f65d80fd190 2024-06-18T03:03:03.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:03.502+0000 7f65ceffd640 1 -- 172.21.15.16:0/3621386435 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f65c0054170 con 0x7f65d80fd190 2024-06-18T03:03:03.502 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T03:03:03.502 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-18T03:03:03.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:03.504+0000 7f65dccfa640 1 -- 172.21.15.16:0/3621386435 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f65c4065430 msgr2=0x7f65c40678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:03:03.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:03.504+0000 7f65dccfa640 1 --2- 172.21.15.16:0/3621386435 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f65c4065430 0x7f65c40678e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f65c8009f80 tx=0x7f65c8035040 comp rx=0 tx=0).stop 2024-06-18T03:03:03.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:03.504+0000 7f65dccfa640 1 -- 172.21.15.16:0/3621386435 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65d80fd190 msgr2=0x7f65d8113590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:03:03.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:03.504+0000 7f65dccfa640 1 --2- 172.21.15.16:0/3621386435 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65d80fd190 0x7f65d8113590 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f65c000c790 tx=0x7f65c00043c0 comp rx=0 tx=0).stop 2024-06-18T03:03:03.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:03.504+0000 7f65dccfa640 1 -- 172.21.15.16:0/3621386435 shutdown_connections 2024-06-18T03:03:03.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:03.504+0000 7f65dccfa640 1 --2- 172.21.15.16:0/3621386435 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f65c4065430 0x7f65c40678e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:03:03.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:03.504+0000 7f65dccfa640 1 --2- 172.21.15.16:0/3621386435 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f65d80fd190 0x7f65d8113590 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:03:03.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:03.504+0000 7f65dccfa640 1 --2- 172.21.15.16:0/3621386435 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f65d80fba50 0x7f65d8113050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:03:03.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:03.504+0000 7f65dccfa640 1 -- 172.21.15.16:0/3621386435 >> 172.21.15.16:0/3621386435 conn(0x7f65d8071860 msgr2=0x7f65d80fb1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:03:03.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:03.504+0000 7f65dccfa640 1 -- 172.21.15.16:0/3621386435 shutdown_connections 2024-06-18T03:03:03.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:03.504+0000 7f65dccfa640 1 -- 172.21.15.16:0/3621386435 wait complete. 2024-06-18T03:03:03.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:03:03 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/3621386435' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:03:03.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:03:03 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/3621386435' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:03:04.805 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:03:04 smithi016 ceph-mon[29214]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:03:04.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:03:04 smithi123 ceph-mon[36358]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:03:06.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:03:06 smithi123 ceph-mon[36358]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:03:07.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:03:06 smithi016 ceph-mon[29214]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:03:07.229 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T03:03:07.512 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T03:03:08.872 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:03:08 smithi016 ceph-mon[29214]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:03:08.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:03:08 smithi123 ceph-mon[36358]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:03:09.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:09.201+0000 7fd025f05640 1 -- 172.21.15.16:0/2116319173 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd020102c40 msgr2=0x7fd0200fc2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:03:09.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:09.201+0000 7fd025f05640 1 --2- 172.21.15.16:0/2116319173 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd020102c40 0x7fd0200fc2d0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fd0140048d0 tx=0x7fd014009da0 comp rx=0 tx=0).stop 2024-06-18T03:03:09.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:09.201+0000 7fd025f05640 1 -- 172.21.15.16:0/2116319173 shutdown_connections 2024-06-18T03:03:09.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:09.201+0000 7fd025f05640 1 --2- 172.21.15.16:0/2116319173 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd020102c40 0x7fd0200fc2d0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:03:09.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:09.201+0000 7fd025f05640 1 --2- 172.21.15.16:0/2116319173 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd0201022a0 0x7fd020102670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:03:09.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:09.201+0000 7fd025f05640 1 -- 172.21.15.16:0/2116319173 >> 172.21.15.16:0/2116319173 conn(0x7fd0200f7e70 msgr2=0x7fd0200fa280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:03:09.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:09.201+0000 7fd025f05640 1 -- 172.21.15.16:0/2116319173 shutdown_connections 2024-06-18T03:03:09.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:09.201+0000 7fd025f05640 1 -- 172.21.15.16:0/2116319173 wait complete. 2024-06-18T03:03:09.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:09.202+0000 7fd025f05640 1 Processor -- start 2024-06-18T03:03:09.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:09.202+0000 7fd025f05640 1 -- start start 2024-06-18T03:03:09.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:09.202+0000 7fd025f05640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0201022a0 0x7fd020113000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:03:09.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:09.202+0000 7fd025f05640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd020102c40 0x7fd020113540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:03:09.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:09.202+0000 7fd025f05640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd020113b40 con 0x7fd0201022a0 2024-06-18T03:03:09.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:09.202+0000 7fd025f05640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd02010cf80 con 0x7fd020102c40 2024-06-18T03:03:09.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:09.202+0000 7fd01f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0201022a0 0x7fd020113000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:03:09.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:09.202+0000 7fd01f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0201022a0 0x7fd020113000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36604/0 (socket says 172.21.15.16:36604) 2024-06-18T03:03:09.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:09.202+0000 7fd01f7fe640 1 -- 172.21.15.16:0/4196063342 learned_addr learned my addr 172.21.15.16:0/4196063342 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T03:03:09.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:09.202+0000 7fd01effd640 1 --2- 172.21.15.16:0/4196063342 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd020102c40 0x7fd020113540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:03:09.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:09.203+0000 7fd01f7fe640 1 -- 172.21.15.16:0/4196063342 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd020102c40 msgr2=0x7fd020113540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:03:09.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:09.203+0000 7fd01f7fe640 1 --2- 172.21.15.16:0/4196063342 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd020102c40 0x7fd020113540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:03:09.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:09.203+0000 7fd01f7fe640 1 -- 172.21.15.16:0/4196063342 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd014005040 con 0x7fd0201022a0 2024-06-18T03:03:09.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:09.203+0000 7fd01f7fe640 1 --2- 172.21.15.16:0/4196063342 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0201022a0 0x7fd020113000 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fd01000c810 tx=0x7fd0100043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:03:09.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:09.203+0000 7fd007fff640 1 -- 172.21.15.16:0/4196063342 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd01000a820 con 0x7fd0201022a0 2024-06-18T03:03:09.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:09.203+0000 7fd025f05640 1 -- 172.21.15.16:0/4196063342 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd02010d260 con 0x7fd0201022a0 2024-06-18T03:03:09.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:09.203+0000 7fd007fff640 1 -- 172.21.15.16:0/4196063342 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd01000ae50 con 0x7fd0201022a0 2024-06-18T03:03:09.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:09.203+0000 7fd007fff640 1 -- 172.21.15.16:0/4196063342 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd0100115c0 con 0x7fd0201022a0 2024-06-18T03:03:09.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:09.203+0000 7fd025f05640 1 -- 172.21.15.16:0/4196063342 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd02010d800 con 0x7fd0201022a0 2024-06-18T03:03:09.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:09.204+0000 7fd025f05640 1 -- 172.21.15.16:0/4196063342 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0200fda10 con 0x7fd0201022a0 2024-06-18T03:03:09.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:09.204+0000 7fd007fff640 1 -- 172.21.15.16:0/4196063342 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fd01000a980 con 0x7fd0201022a0 2024-06-18T03:03:09.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:09.205+0000 7fd007fff640 1 --2- 172.21.15.16:0/4196063342 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fd008065430 0x7fd0080678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:03:09.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:09.205+0000 7fd007fff640 1 -- 172.21.15.16:0/4196063342 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fd010082b00 con 0x7fd0201022a0 2024-06-18T03:03:09.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:09.205+0000 7fd01effd640 1 --2- 172.21.15.16:0/4196063342 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fd008065430 0x7fd0080678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:03:09.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:09.205+0000 7fd01effd640 1 --2- 172.21.15.16:0/4196063342 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fd008065430 0x7fd0080678e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd02010e8c0 tx=0x7fd014035040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:03:09.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:09.209+0000 7fd007fff640 1 -- 172.21.15.16:0/4196063342 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fd010050f10 con 0x7fd0201022a0 2024-06-18T03:03:09.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:09.419+0000 7fd025f05640 1 -- 172.21.15.16:0/4196063342 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd0201093e0 con 0x7fd0201022a0 2024-06-18T03:03:09.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:09.419+0000 7fd007fff640 1 -- 172.21.15.16:0/4196063342 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fd010054530 con 0x7fd0201022a0 2024-06-18T03:03:09.420 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T03:03:09.420 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-18T03:03:09.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:09.422+0000 7fd025f05640 1 -- 172.21.15.16:0/4196063342 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fd008065430 msgr2=0x7fd0080678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:03:09.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:09.422+0000 7fd025f05640 1 --2- 172.21.15.16:0/4196063342 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fd008065430 0x7fd0080678e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd02010e8c0 tx=0x7fd014035040 comp rx=0 tx=0).stop 2024-06-18T03:03:09.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:09.422+0000 7fd025f05640 1 -- 172.21.15.16:0/4196063342 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0201022a0 msgr2=0x7fd020113000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:03:09.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:09.422+0000 7fd025f05640 1 --2- 172.21.15.16:0/4196063342 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0201022a0 0x7fd020113000 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fd01000c810 tx=0x7fd0100043c0 comp rx=0 tx=0).stop 2024-06-18T03:03:09.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:09.422+0000 7fd025f05640 1 -- 172.21.15.16:0/4196063342 shutdown_connections 2024-06-18T03:03:09.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:09.422+0000 7fd025f05640 1 --2- 172.21.15.16:0/4196063342 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fd008065430 0x7fd0080678e0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:03:09.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:09.422+0000 7fd025f05640 1 --2- 172.21.15.16:0/4196063342 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd020102c40 0x7fd020113540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:03:09.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:09.422+0000 7fd025f05640 1 --2- 172.21.15.16:0/4196063342 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0201022a0 0x7fd020113000 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:03:09.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:09.422+0000 7fd025f05640 1 -- 172.21.15.16:0/4196063342 >> 172.21.15.16:0/4196063342 conn(0x7fd0200f7e70 msgr2=0x7fd0200f8e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:03:09.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:09.422+0000 7fd025f05640 1 -- 172.21.15.16:0/4196063342 shutdown_connections 2024-06-18T03:03:09.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:09.422+0000 7fd025f05640 1 -- 172.21.15.16:0/4196063342 wait complete. 2024-06-18T03:03:09.791 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:03:09 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/4196063342' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:03:09.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:03:09 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/4196063342' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:03:10.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:03:10 smithi123 ceph-mon[36358]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:03:11.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:03:10 smithi016 ceph-mon[29214]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:03:11.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:03:11 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T03:03:12.055 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:03:11 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T03:03:12.956 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:03:12 smithi016 ceph-mon[29214]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:03:12.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:03:12 smithi123 ceph-mon[36358]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:03:13.204 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T03:03:13.483 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T03:03:13.870 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:03:13 smithi016 ceph-mon[29214]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:03:13.984 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:03:13 smithi123 ceph-mon[36358]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:03:15.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:15.203+0000 7f9938fc3640 1 -- 172.21.15.16:0/4264277134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9934102c60 msgr2=0x7f99340fc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:03:15.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:15.203+0000 7f9938fc3640 1 --2- 172.21.15.16:0/4264277134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9934102c60 0x7f99340fc2f0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f991c0048d0 tx=0x7f991c009da0 comp rx=0 tx=0).stop 2024-06-18T03:03:15.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:15.203+0000 7f9938fc3640 1 -- 172.21.15.16:0/4264277134 shutdown_connections 2024-06-18T03:03:15.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:15.203+0000 7f9938fc3640 1 --2- 172.21.15.16:0/4264277134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9934102c60 0x7f99340fc2f0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:03:15.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:15.203+0000 7f9938fc3640 1 --2- 172.21.15.16:0/4264277134 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99341022c0 0x7f9934102690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:03:15.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:15.203+0000 7f9938fc3640 1 -- 172.21.15.16:0/4264277134 >> 172.21.15.16:0/4264277134 conn(0x7f99340f7e50 msgr2=0x7f99340fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:03:15.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:15.203+0000 7f9938fc3640 1 -- 172.21.15.16:0/4264277134 shutdown_connections 2024-06-18T03:03:15.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:15.203+0000 7f9938fc3640 1 -- 172.21.15.16:0/4264277134 wait complete. 2024-06-18T03:03:15.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:15.204+0000 7f9938fc3640 1 Processor -- start 2024-06-18T03:03:15.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:15.204+0000 7f9938fc3640 1 -- start start 2024-06-18T03:03:15.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:15.204+0000 7f9938fc3640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99341022c0 0x7f9934113370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:03:15.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:15.204+0000 7f9938fc3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9934102c60 0x7f99341138b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:03:15.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:15.204+0000 7f9938fc3640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9934113eb0 con 0x7f9934102c60 2024-06-18T03:03:15.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:15.204+0000 7f9938fc3640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f993410d2f0 con 0x7f99341022c0 2024-06-18T03:03:15.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:15.204+0000 7f992bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9934102c60 0x7f99341138b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:03:15.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:15.204+0000 7f992bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9934102c60 0x7f99341138b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38808/0 (socket says 172.21.15.16:38808) 2024-06-18T03:03:15.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:15.204+0000 7f992bfff640 1 -- 172.21.15.16:0/2692455386 learned_addr learned my addr 172.21.15.16:0/2692455386 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T03:03:15.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:15.204+0000 7f993259c640 1 --2- 172.21.15.16:0/2692455386 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99341022c0 0x7f9934113370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:03:15.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:15.205+0000 7f992bfff640 1 -- 172.21.15.16:0/2692455386 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99341022c0 msgr2=0x7f9934113370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:03:15.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:15.205+0000 7f992bfff640 1 --2- 172.21.15.16:0/2692455386 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99341022c0 0x7f9934113370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:03:15.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:15.205+0000 7f992bfff640 1 -- 172.21.15.16:0/2692455386 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f991c005040 con 0x7f9934102c60 2024-06-18T03:03:15.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:15.205+0000 7f992bfff640 1 --2- 172.21.15.16:0/2692455386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9934102c60 0x7f99341138b0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f991c003080 tx=0x7f991c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:03:15.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:15.205+0000 7f992affd640 1 -- 172.21.15.16:0/2692455386 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f991c038070 con 0x7f9934102c60 2024-06-18T03:03:15.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:15.205+0000 7f992affd640 1 -- 172.21.15.16:0/2692455386 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f991c008ee0 con 0x7f9934102c60 2024-06-18T03:03:15.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:15.205+0000 7f992affd640 1 -- 172.21.15.16:0/2692455386 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f991c03d870 con 0x7f9934102c60 2024-06-18T03:03:15.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:15.205+0000 7f9938fc3640 1 -- 172.21.15.16:0/2692455386 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f993410d570 con 0x7f9934102c60 2024-06-18T03:03:15.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:15.205+0000 7f9938fc3640 1 -- 172.21.15.16:0/2692455386 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f993410dab0 con 0x7f9934102c60 2024-06-18T03:03:15.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:15.206+0000 7f9938fc3640 1 -- 172.21.15.16:0/2692455386 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99340fda30 con 0x7f9934102c60 2024-06-18T03:03:15.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:15.207+0000 7f992affd640 1 -- 172.21.15.16:0/2692455386 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f991c030020 con 0x7f9934102c60 2024-06-18T03:03:15.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:15.207+0000 7f992affd640 1 --2- 172.21.15.16:0/2692455386 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f9920065430 0x7f99200678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:03:15.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:15.207+0000 7f992affd640 1 -- 172.21.15.16:0/2692455386 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f991c0a21a0 con 0x7f9934102c60 2024-06-18T03:03:15.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:15.207+0000 7f993259c640 1 --2- 172.21.15.16:0/2692455386 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f9920065430 0x7f99200678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:03:15.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:15.208+0000 7f993259c640 1 --2- 172.21.15.16:0/2692455386 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f9920065430 0x7f99200678e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f99240041d0 tx=0x7f9924004330 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:03:15.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:15.213+0000 7f992affd640 1 -- 172.21.15.16:0/2692455386 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f991c0706d0 con 0x7f9934102c60 2024-06-18T03:03:15.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:15.420+0000 7f9938fc3640 1 -- 172.21.15.16:0/2692455386 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f9934109420 con 0x7f9934102c60 2024-06-18T03:03:15.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:15.421+0000 7f992affd640 1 -- 172.21.15.16:0/2692455386 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f991c073cf0 con 0x7f9934102c60 2024-06-18T03:03:15.421 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T03:03:15.422 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"2 mgr modules have recently crashed","count":2},"muted":false}},"mutes":[]} 2024-06-18T03:03:15.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:15.423+0000 7f9938fc3640 1 -- 172.21.15.16:0/2692455386 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f9920065430 msgr2=0x7f99200678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:03:15.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:15.423+0000 7f9938fc3640 1 --2- 172.21.15.16:0/2692455386 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f9920065430 0x7f99200678e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f99240041d0 tx=0x7f9924004330 comp rx=0 tx=0).stop 2024-06-18T03:03:15.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:15.423+0000 7f9938fc3640 1 -- 172.21.15.16:0/2692455386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9934102c60 msgr2=0x7f99341138b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:03:15.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:15.423+0000 7f9938fc3640 1 --2- 172.21.15.16:0/2692455386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9934102c60 0x7f99341138b0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f991c003080 tx=0x7f991c009da0 comp rx=0 tx=0).stop 2024-06-18T03:03:15.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:15.423+0000 7f9938fc3640 1 -- 172.21.15.16:0/2692455386 shutdown_connections 2024-06-18T03:03:15.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:15.423+0000 7f9938fc3640 1 --2- 172.21.15.16:0/2692455386 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7f9920065430 0x7f99200678e0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:03:15.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:15.423+0000 7f9938fc3640 1 --2- 172.21.15.16:0/2692455386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9934102c60 0x7f99341138b0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:03:15.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:15.423+0000 7f9938fc3640 1 --2- 172.21.15.16:0/2692455386 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99341022c0 0x7f9934113370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:03:15.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:15.423+0000 7f9938fc3640 1 -- 172.21.15.16:0/2692455386 >> 172.21.15.16:0/2692455386 conn(0x7f99340f7e50 msgr2=0x7f99340f8db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:03:15.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:15.423+0000 7f9938fc3640 1 -- 172.21.15.16:0/2692455386 shutdown_connections 2024-06-18T03:03:15.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:15.423+0000 7f9938fc3640 1 -- 172.21.15.16:0/2692455386 wait complete. 2024-06-18T03:03:16.484 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:03:16 smithi123 ceph-mon[36358]: from='client.14626 -' entity='client.crash.smithi016' cmd=[{"prefix": "crash post", "target": ["mon-mgr", ""]}]: dispatch 2024-06-18T03:03:16.485 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:03:16 smithi123 ceph-mon[36358]: from='client.14630 -' entity='client.crash.smithi016' cmd=[{"prefix": "crash post", "target": ["mon-mgr", ""]}]: dispatch 2024-06-18T03:03:16.485 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:03:16 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T03:03:16.485 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:03:16 smithi123 ceph-mon[36358]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:03:16.485 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:03:16 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/2692455386' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:03:16.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:03:16 smithi016 ceph-mon[29214]: from='client.14626 -' entity='client.crash.smithi016' cmd=[{"prefix": "crash post", "target": ["mon-mgr", ""]}]: dispatch 2024-06-18T03:03:16.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:03:16 smithi016 ceph-mon[29214]: from='client.14630 -' entity='client.crash.smithi016' cmd=[{"prefix": "crash post", "target": ["mon-mgr", ""]}]: dispatch 2024-06-18T03:03:16.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:03:16 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' 2024-06-18T03:03:16.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:03:16 smithi016 ceph-mon[29214]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:03:16.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:03:16 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/2692455386' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:03:17.484 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:03:17 smithi123 ceph-mon[36358]: Health check update: 3 mgr modules have recently crashed (RECENT_MGR_MODULE_CRASH) 2024-06-18T03:03:17.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:03:17 smithi016 ceph-mon[29214]: Health check update: 3 mgr modules have recently crashed (RECENT_MGR_MODULE_CRASH) 2024-06-18T03:03:18.484 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:03:18 smithi123 ceph-mon[36358]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:03:18.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:03:18 smithi016 ceph-mon[29214]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:03:19.157 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T03:03:19.436 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T03:03:20.484 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:03:20 smithi123 ceph-mon[36358]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:03:20.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:03:20 smithi016 ceph-mon[29214]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:03:21.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:21.109+0000 7fa7a7710640 1 -- 172.21.15.16:0/2145360916 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7a00fdcb0 msgr2=0x7fa7a01062f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:03:21.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:21.109+0000 7fa7a7710640 1 --2- 172.21.15.16:0/2145360916 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7a00fdcb0 0x7fa7a01062f0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fa790004840 tx=0x7fa790009da0 comp rx=0 tx=0).stop 2024-06-18T03:03:21.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:21.109+0000 7fa7a7710640 1 -- 172.21.15.16:0/2145360916 shutdown_connections 2024-06-18T03:03:21.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:21.109+0000 7fa7a7710640 1 --2- 172.21.15.16:0/2145360916 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7a00fdcb0 0x7fa7a01062f0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:03:21.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:21.109+0000 7fa7a7710640 1 --2- 172.21.15.16:0/2145360916 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa7a00fd310 0x7fa7a00fd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:03:21.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:21.109+0000 7fa7a7710640 1 -- 172.21.15.16:0/2145360916 >> 172.21.15.16:0/2145360916 conn(0x7fa7a00f6d60 msgr2=0x7fa7a00f9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:03:21.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:21.109+0000 7fa7a7710640 1 -- 172.21.15.16:0/2145360916 shutdown_connections 2024-06-18T03:03:21.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:21.110+0000 7fa7a7710640 1 -- 172.21.15.16:0/2145360916 wait complete. 2024-06-18T03:03:21.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:21.110+0000 7fa7a7710640 1 Processor -- start 2024-06-18T03:03:21.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:21.110+0000 7fa7a7710640 1 -- start start 2024-06-18T03:03:21.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:21.110+0000 7fa7a7710640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa7a00fd310 0x7fa7a00fb140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:03:21.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:21.110+0000 7fa7a7710640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7a00fdcb0 0x7fa7a00fb680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:03:21.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:21.110+0000 7fa7a7710640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a00fbbc0 con 0x7fa7a00fdcb0 2024-06-18T03:03:21.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:21.110+0000 7fa7a7710640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a00fbd30 con 0x7fa7a00fd310 2024-06-18T03:03:21.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:21.110+0000 7fa7a4cab640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7a00fdcb0 0x7fa7a00fb680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:03:21.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:21.110+0000 7fa7a4cab640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7a00fdcb0 0x7fa7a00fb680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46770/0 (socket says 172.21.15.16:46770) 2024-06-18T03:03:21.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:21.110+0000 7fa7a4cab640 1 -- 172.21.15.16:0/3918581666 learned_addr learned my addr 172.21.15.16:0/3918581666 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T03:03:21.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:21.110+0000 7fa7a54ac640 1 --2- 172.21.15.16:0/3918581666 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa7a00fd310 0x7fa7a00fb140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:03:21.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:21.111+0000 7fa7a4cab640 1 -- 172.21.15.16:0/3918581666 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa7a00fd310 msgr2=0x7fa7a00fb140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:03:21.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:21.111+0000 7fa7a4cab640 1 --2- 172.21.15.16:0/3918581666 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa7a00fd310 0x7fa7a00fb140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:03:21.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:21.111+0000 7fa7a4cab640 1 -- 172.21.15.16:0/3918581666 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa790005040 con 0x7fa7a00fdcb0 2024-06-18T03:03:21.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:21.111+0000 7fa7a4cab640 1 --2- 172.21.15.16:0/3918581666 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7a00fdcb0 0x7fa7a00fb680 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fa79002a570 tx=0x7fa790009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:03:21.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:21.111+0000 7fa795ffb640 1 -- 172.21.15.16:0/3918581666 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa790038070 con 0x7fa7a00fdcb0 2024-06-18T03:03:21.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:21.111+0000 7fa795ffb640 1 -- 172.21.15.16:0/3918581666 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa790032880 con 0x7fa7a00fdcb0 2024-06-18T03:03:21.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:21.111+0000 7fa7a7710640 1 -- 172.21.15.16:0/3918581666 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7a00fbf60 con 0x7fa7a00fdcb0 2024-06-18T03:03:21.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:21.111+0000 7fa795ffb640 1 -- 172.21.15.16:0/3918581666 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa79003d780 con 0x7fa7a00fdcb0 2024-06-18T03:03:21.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:21.111+0000 7fa7a7710640 1 -- 172.21.15.16:0/3918581666 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7a0119630 con 0x7fa7a00fdcb0 2024-06-18T03:03:21.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:21.112+0000 7fa7a7710640 1 -- 172.21.15.16:0/3918581666 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7a0066050 con 0x7fa7a00fdcb0 2024-06-18T03:03:21.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:21.113+0000 7fa795ffb640 1 -- 172.21.15.16:0/3918581666 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fa790030020 con 0x7fa7a00fdcb0 2024-06-18T03:03:21.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:21.113+0000 7fa795ffb640 1 --2- 172.21.15.16:0/3918581666 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fa78c065430 0x7fa78c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:03:21.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:21.113+0000 7fa795ffb640 1 -- 172.21.15.16:0/3918581666 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fa7900a28a0 con 0x7fa7a00fdcb0 2024-06-18T03:03:21.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:21.113+0000 7fa7a54ac640 1 --2- 172.21.15.16:0/3918581666 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fa78c065430 0x7fa78c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:03:21.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:21.114+0000 7fa7a54ac640 1 --2- 172.21.15.16:0/3918581666 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fa78c065430 0x7fa78c0678e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa79c009af0 tx=0x7fa79c00b040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:03:21.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:21.118+0000 7fa795ffb640 1 -- 172.21.15.16:0/3918581666 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fa790070d50 con 0x7fa7a00fdcb0 2024-06-18T03:03:21.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:21.326+0000 7fa7a7710640 1 -- 172.21.15.16:0/3918581666 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fa7a00fa9a0 con 0x7fa7a00fdcb0 2024-06-18T03:03:21.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:21.327+0000 7fa795ffb640 1 -- 172.21.15.16:0/3918581666 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fa790074370 con 0x7fa7a00fdcb0 2024-06-18T03:03:21.327 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T03:03:21.327 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-18T03:03:21.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:21.328+0000 7fa7a7710640 1 -- 172.21.15.16:0/3918581666 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fa78c065430 msgr2=0x7fa78c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:03:21.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:21.328+0000 7fa7a7710640 1 --2- 172.21.15.16:0/3918581666 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fa78c065430 0x7fa78c0678e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa79c009af0 tx=0x7fa79c00b040 comp rx=0 tx=0).stop 2024-06-18T03:03:21.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:21.328+0000 7fa7a7710640 1 -- 172.21.15.16:0/3918581666 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7a00fdcb0 msgr2=0x7fa7a00fb680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:03:21.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:21.328+0000 7fa7a7710640 1 --2- 172.21.15.16:0/3918581666 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7a00fdcb0 0x7fa7a00fb680 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fa79002a570 tx=0x7fa790009da0 comp rx=0 tx=0).stop 2024-06-18T03:03:21.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:21.328+0000 7fa7a7710640 1 -- 172.21.15.16:0/3918581666 shutdown_connections 2024-06-18T03:03:21.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:21.328+0000 7fa7a7710640 1 --2- 172.21.15.16:0/3918581666 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fa78c065430 0x7fa78c0678e0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:03:21.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:21.329+0000 7fa7a7710640 1 --2- 172.21.15.16:0/3918581666 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa7a00fdcb0 0x7fa7a00fb680 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:03:21.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:21.329+0000 7fa7a7710640 1 --2- 172.21.15.16:0/3918581666 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa7a00fd310 0x7fa7a00fb140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:03:21.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:21.329+0000 7fa7a7710640 1 -- 172.21.15.16:0/3918581666 >> 172.21.15.16:0/3918581666 conn(0x7fa7a00f6d60 msgr2=0x7fa7a0100b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:03:21.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:21.329+0000 7fa7a7710640 1 -- 172.21.15.16:0/3918581666 shutdown_connections 2024-06-18T03:03:21.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:21.329+0000 7fa7a7710640 1 -- 172.21.15.16:0/3918581666 wait complete. 2024-06-18T03:03:22.484 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:03:22 smithi123 ceph-mon[36358]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:03:22.484 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:03:22 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/3918581666' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:03:22.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:03:22 smithi016 ceph-mon[29214]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:03:22.556 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:03:22 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/3918581666' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:03:24.484 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:03:24 smithi123 ceph-mon[36358]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:03:24.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:03:24 smithi016 ceph-mon[29214]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:03:25.090 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T03:03:25.367 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T03:03:26.484 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:03:26 smithi123 ceph-mon[36358]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:03:26.502 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:03:26 smithi016 ceph-mon[29214]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:03:27.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:27.066+0000 7fb3845de640 1 -- 172.21.15.16:0/3611304782 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb37c1022f0 msgr2=0x7fb37c1026c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:03:27.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:27.066+0000 7fb3845de640 1 --2- 172.21.15.16:0/3611304782 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb37c1022f0 0x7fb37c1026c0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fb3780048d0 tx=0x7fb378009da0 comp rx=0 tx=0).stop 2024-06-18T03:03:27.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:27.066+0000 7fb3845de640 1 -- 172.21.15.16:0/3611304782 shutdown_connections 2024-06-18T03:03:27.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:27.066+0000 7fb3845de640 1 --2- 172.21.15.16:0/3611304782 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb37c102c90 0x7fb37c0fa9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:03:27.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:27.066+0000 7fb3845de640 1 --2- 172.21.15.16:0/3611304782 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb37c1022f0 0x7fb37c1026c0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:03:27.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:27.066+0000 7fb3845de640 1 -- 172.21.15.16:0/3611304782 >> 172.21.15.16:0/3611304782 conn(0x7fb37c070a40 msgr2=0x7fb37c070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:03:27.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:27.066+0000 7fb3845de640 1 -- 172.21.15.16:0/3611304782 shutdown_connections 2024-06-18T03:03:27.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:27.066+0000 7fb3845de640 1 -- 172.21.15.16:0/3611304782 wait complete. 2024-06-18T03:03:27.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:27.067+0000 7fb3845de640 1 Processor -- start 2024-06-18T03:03:27.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:27.067+0000 7fb3845de640 1 -- start start 2024-06-18T03:03:27.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:27.067+0000 7fb3845de640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb37c1022f0 0x7fb37c1130a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:03:27.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:27.067+0000 7fb3845de640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb37c102c90 0x7fb37c1135e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:03:27.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:27.067+0000 7fb3845de640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb37c113be0 con 0x7fb37c102c90 2024-06-18T03:03:27.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:27.067+0000 7fb3845de640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb37c10d020 con 0x7fb37c1022f0 2024-06-18T03:03:27.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:27.067+0000 7fb381b79640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb37c102c90 0x7fb37c1135e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:03:27.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:27.067+0000 7fb381b79640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb37c102c90 0x7fb37c1135e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46786/0 (socket says 172.21.15.16:46786) 2024-06-18T03:03:27.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:27.067+0000 7fb381b79640 1 -- 172.21.15.16:0/2272761362 learned_addr learned my addr 172.21.15.16:0/2272761362 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T03:03:27.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:27.067+0000 7fb38237a640 1 --2- 172.21.15.16:0/2272761362 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb37c1022f0 0x7fb37c1130a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:03:27.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:27.067+0000 7fb381b79640 1 -- 172.21.15.16:0/2272761362 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb37c1022f0 msgr2=0x7fb37c1130a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:03:27.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:27.067+0000 7fb381b79640 1 --2- 172.21.15.16:0/2272761362 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb37c1022f0 0x7fb37c1130a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:03:27.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:27.067+0000 7fb381b79640 1 -- 172.21.15.16:0/2272761362 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb378005040 con 0x7fb37c102c90 2024-06-18T03:03:27.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:27.068+0000 7fb381b79640 1 --2- 172.21.15.16:0/2272761362 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb37c102c90 0x7fb37c1135e0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fb36c00c760 tx=0x7fb36c00cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:03:27.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:27.068+0000 7fb372ffd640 1 -- 172.21.15.16:0/2272761362 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb36c00a7a0 con 0x7fb37c102c90 2024-06-18T03:03:27.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:27.068+0000 7fb372ffd640 1 -- 172.21.15.16:0/2272761362 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb36c00add0 con 0x7fb37c102c90 2024-06-18T03:03:27.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:27.068+0000 7fb3845de640 1 -- 172.21.15.16:0/2272761362 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb37c10d300 con 0x7fb37c102c90 2024-06-18T03:03:27.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:27.068+0000 7fb372ffd640 1 -- 172.21.15.16:0/2272761362 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb36c0113a0 con 0x7fb37c102c90 2024-06-18T03:03:27.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:27.068+0000 7fb3845de640 1 -- 172.21.15.16:0/2272761362 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb37c10d820 con 0x7fb37c102c90 2024-06-18T03:03:27.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:27.069+0000 7fb3845de640 1 -- 172.21.15.16:0/2272761362 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb37c065510 con 0x7fb37c102c90 2024-06-18T03:03:27.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:27.069+0000 7fb372ffd640 1 -- 172.21.15.16:0/2272761362 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fb36c012400 con 0x7fb37c102c90 2024-06-18T03:03:27.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:27.070+0000 7fb372ffd640 1 --2- 172.21.15.16:0/2272761362 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fb368065430 0x7fb3680678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:03:27.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:27.070+0000 7fb372ffd640 1 -- 172.21.15.16:0/2272761362 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fb36c0825a0 con 0x7fb37c102c90 2024-06-18T03:03:27.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:27.070+0000 7fb38237a640 1 --2- 172.21.15.16:0/2272761362 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fb368065430 0x7fb3680678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:03:27.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:27.070+0000 7fb38237a640 1 --2- 172.21.15.16:0/2272761362 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fb368065430 0x7fb3680678e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb378003080 tx=0x7fb378035040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:03:27.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:27.074+0000 7fb372ffd640 1 -- 172.21.15.16:0/2272761362 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fb36c050ad0 con 0x7fb37c102c90 2024-06-18T03:03:27.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:27.285+0000 7fb3845de640 1 -- 172.21.15.16:0/2272761362 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fb37c074960 con 0x7fb37c102c90 2024-06-18T03:03:27.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:27.286+0000 7fb372ffd640 1 -- 172.21.15.16:0/2272761362 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fb36c0540f0 con 0x7fb37c102c90 2024-06-18T03:03:27.286 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T03:03:27.287 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-18T03:03:27.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:27.288+0000 7fb3845de640 1 -- 172.21.15.16:0/2272761362 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fb368065430 msgr2=0x7fb3680678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:03:27.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:27.288+0000 7fb3845de640 1 --2- 172.21.15.16:0/2272761362 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fb368065430 0x7fb3680678e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb378003080 tx=0x7fb378035040 comp rx=0 tx=0).stop 2024-06-18T03:03:27.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:27.288+0000 7fb3845de640 1 -- 172.21.15.16:0/2272761362 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb37c102c90 msgr2=0x7fb37c1135e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:03:27.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:27.288+0000 7fb3845de640 1 --2- 172.21.15.16:0/2272761362 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb37c102c90 0x7fb37c1135e0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fb36c00c760 tx=0x7fb36c00cb60 comp rx=0 tx=0).stop 2024-06-18T03:03:27.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:27.288+0000 7fb3845de640 1 -- 172.21.15.16:0/2272761362 shutdown_connections 2024-06-18T03:03:27.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:27.288+0000 7fb3845de640 1 --2- 172.21.15.16:0/2272761362 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fb368065430 0x7fb3680678e0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:03:27.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:27.288+0000 7fb3845de640 1 --2- 172.21.15.16:0/2272761362 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb37c102c90 0x7fb37c1135e0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:03:27.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:27.288+0000 7fb3845de640 1 --2- 172.21.15.16:0/2272761362 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb37c1022f0 0x7fb37c1130a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:03:27.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:27.288+0000 7fb3845de640 1 -- 172.21.15.16:0/2272761362 >> 172.21.15.16:0/2272761362 conn(0x7fb37c070a40 msgr2=0x7fb37c1050b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:03:27.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:27.288+0000 7fb3845de640 1 -- 172.21.15.16:0/2272761362 shutdown_connections 2024-06-18T03:03:27.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:27.288+0000 7fb3845de640 1 -- 172.21.15.16:0/2272761362 wait complete. 2024-06-18T03:03:27.484 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:03:27 smithi123 ceph-mon[36358]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T03:03:27.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:03:27 smithi016 ceph-mon[29214]: from='mgr.14216 172.21.15.16:0/1695864429' entity='mgr.smithi016.amxydz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-18T03:03:28.484 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:03:28 smithi123 ceph-mon[36358]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:03:28.485 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:03:28 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/2272761362' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:03:28.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:03:28 smithi016 ceph-mon[29214]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:03:28.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:03:28 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/2272761362' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:03:30.484 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:03:30 smithi123 ceph-mon[36358]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:03:30.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:03:30 smithi016 ceph-mon[29214]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:03:30.929 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-18T03:03:31.207 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/mon.smithi016/config 2024-06-18T03:03:32.484 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:03:32 smithi123 ceph-mon[36358]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:03:32.512 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:03:32 smithi016 ceph-mon[29214]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:03:33.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:33.032+0000 7fe170ed8640 1 -- 172.21.15.16:0/1866660495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe16c1022c0 msgr2=0x7fe16c102690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:03:33.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:33.032+0000 7fe170ed8640 1 --2- 172.21.15.16:0/1866660495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe16c1022c0 0x7fe16c102690 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fe15c0048d0 tx=0x7fe15c009da0 comp rx=0 tx=0).stop 2024-06-18T03:03:33.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:33.032+0000 7fe170ed8640 1 -- 172.21.15.16:0/1866660495 shutdown_connections 2024-06-18T03:03:33.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:33.032+0000 7fe170ed8640 1 --2- 172.21.15.16:0/1866660495 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe16c102c60 0x7fe16c0fc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:03:33.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:33.032+0000 7fe170ed8640 1 --2- 172.21.15.16:0/1866660495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe16c1022c0 0x7fe16c102690 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:03:33.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:33.032+0000 7fe170ed8640 1 -- 172.21.15.16:0/1866660495 >> 172.21.15.16:0/1866660495 conn(0x7fe16c0f7e50 msgr2=0x7fe16c0fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:03:33.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:33.032+0000 7fe170ed8640 1 -- 172.21.15.16:0/1866660495 shutdown_connections 2024-06-18T03:03:33.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:33.032+0000 7fe170ed8640 1 -- 172.21.15.16:0/1866660495 wait complete. 2024-06-18T03:03:33.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:33.032+0000 7fe170ed8640 1 Processor -- start 2024-06-18T03:03:33.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:33.032+0000 7fe170ed8640 1 -- start start 2024-06-18T03:03:33.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:33.033+0000 7fe170ed8640 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe16c1022c0 0x7fe16c113330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:03:33.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:33.033+0000 7fe170ed8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe16c102c60 0x7fe16c113870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:03:33.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:33.033+0000 7fe170ed8640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe16c113e70 con 0x7fe16c102c60 2024-06-18T03:03:33.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:33.033+0000 7fe170ed8640 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe16c10d2b0 con 0x7fe16c1022c0 2024-06-18T03:03:33.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:33.033+0000 7fe169d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe16c102c60 0x7fe16c113870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:03:33.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:33.033+0000 7fe169d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe16c102c60 0x7fe16c113870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34212/0 (socket says 172.21.15.16:34212) 2024-06-18T03:03:33.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:33.033+0000 7fe169d9b640 1 -- 172.21.15.16:0/2570885801 learned_addr learned my addr 172.21.15.16:0/2570885801 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-06-18T03:03:33.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:33.033+0000 7fe16a59c640 1 --2- 172.21.15.16:0/2570885801 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe16c1022c0 0x7fe16c113330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:03:33.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:33.033+0000 7fe169d9b640 1 -- 172.21.15.16:0/2570885801 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe16c1022c0 msgr2=0x7fe16c113330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:03:33.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:33.033+0000 7fe169d9b640 1 --2- 172.21.15.16:0/2570885801 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe16c1022c0 0x7fe16c113330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:03:33.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:33.033+0000 7fe169d9b640 1 -- 172.21.15.16:0/2570885801 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe15c005040 con 0x7fe16c102c60 2024-06-18T03:03:33.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:33.033+0000 7fe169d9b640 1 --2- 172.21.15.16:0/2570885801 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe16c102c60 0x7fe16c113870 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fe160002a50 tx=0x7fe160004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:03:33.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:33.033+0000 7fe15affd640 1 -- 172.21.15.16:0/2570885801 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe160007c30 con 0x7fe16c102c60 2024-06-18T03:03:33.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:33.033+0000 7fe15affd640 1 -- 172.21.15.16:0/2570885801 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe16000f040 con 0x7fe16c102c60 2024-06-18T03:03:33.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:33.033+0000 7fe170ed8640 1 -- 172.21.15.16:0/2570885801 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe16c10d590 con 0x7fe16c102c60 2024-06-18T03:03:33.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:33.033+0000 7fe15affd640 1 -- 172.21.15.16:0/2570885801 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe16000dad0 con 0x7fe16c102c60 2024-06-18T03:03:33.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:33.033+0000 7fe170ed8640 1 -- 172.21.15.16:0/2570885801 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe16c10db30 con 0x7fe16c102c60 2024-06-18T03:03:33.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:33.034+0000 7fe170ed8640 1 -- 172.21.15.16:0/2570885801 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe16c0fda30 con 0x7fe16c102c60 2024-06-18T03:03:33.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:33.035+0000 7fe15affd640 1 -- 172.21.15.16:0/2570885801 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fe16000dd40 con 0x7fe16c102c60 2024-06-18T03:03:33.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:33.035+0000 7fe15affd640 1 --2- 172.21.15.16:0/2570885801 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fe154065430 0x7fe1540678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-18T03:03:33.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:33.036+0000 7fe15affd640 1 -- 172.21.15.16:0/2570885801 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fe160081c90 con 0x7fe16c102c60 2024-06-18T03:03:33.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:33.036+0000 7fe16a59c640 1 --2- 172.21.15.16:0/2570885801 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fe154065430 0x7fe1540678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-18T03:03:33.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:33.036+0000 7fe16a59c640 1 --2- 172.21.15.16:0/2570885801 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fe154065430 0x7fe1540678e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fe15c009f80 tx=0x7fe15c035040 comp rx=0 tx=0).ready entity=mgr.14216 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-18T03:03:33.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:33.040+0000 7fe15affd640 1 -- 172.21.15.16:0/2570885801 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fe16004cdf0 con 0x7fe16c102c60 2024-06-18T03:03:33.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:33.254+0000 7fe170ed8640 1 -- 172.21.15.16:0/2570885801 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe16c109420 con 0x7fe16c102c60 2024-06-18T03:03:33.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:33.255+0000 7fe15affd640 1 -- 172.21.15.16:0/2570885801 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fe1600501f0 con 0x7fe16c102c60 2024-06-18T03:03:33.256 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-06-18T03:03:33.256 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-18T03:03:33.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:33.256+0000 7fe170ed8640 1 -- 172.21.15.16:0/2570885801 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fe154065430 msgr2=0x7fe1540678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:03:33.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:33.256+0000 7fe170ed8640 1 --2- 172.21.15.16:0/2570885801 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fe154065430 0x7fe1540678e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fe15c009f80 tx=0x7fe15c035040 comp rx=0 tx=0).stop 2024-06-18T03:03:33.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:33.256+0000 7fe170ed8640 1 -- 172.21.15.16:0/2570885801 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe16c102c60 msgr2=0x7fe16c113870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-18T03:03:33.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:33.256+0000 7fe170ed8640 1 --2- 172.21.15.16:0/2570885801 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe16c102c60 0x7fe16c113870 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fe160002a50 tx=0x7fe160004600 comp rx=0 tx=0).stop 2024-06-18T03:03:33.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:33.257+0000 7fe170ed8640 1 -- 172.21.15.16:0/2570885801 shutdown_connections 2024-06-18T03:03:33.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:33.257+0000 7fe170ed8640 1 --2- 172.21.15.16:0/2570885801 >> [v2:172.21.15.16:6800/3814182425,v1:172.21.15.16:6801/3814182425] conn(0x7fe154065430 0x7fe1540678e0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:03:33.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:33.257+0000 7fe170ed8640 1 --2- 172.21.15.16:0/2570885801 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe16c102c60 0x7fe16c113870 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:03:33.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:33.257+0000 7fe170ed8640 1 --2- 172.21.15.16:0/2570885801 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe16c1022c0 0x7fe16c113330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-18T03:03:33.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:33.257+0000 7fe170ed8640 1 -- 172.21.15.16:0/2570885801 >> 172.21.15.16:0/2570885801 conn(0x7fe16c0f7e50 msgr2=0x7fe16c0f8dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-18T03:03:33.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:33.257+0000 7fe170ed8640 1 -- 172.21.15.16:0/2570885801 shutdown_connections 2024-06-18T03:03:33.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-06-18T03:03:33.257+0000 7fe170ed8640 1 -- 172.21.15.16:0/2570885801 wait complete. 2024-06-18T03:03:34.037 INFO:tasks.cephadm:Teardown begin 2024-06-18T03:03:34.038 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8e9714173de9e92c97e8ef1045d333e96b793454/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph_f18f3f1cbd11e4a41c557e616d172f24deb91473/qa/tasks/cephadm.py", line 2237, in task healthy(ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_ceph_f18f3f1cbd11e4a41c557e616d172f24deb91473/qa/tasks/ceph.py", line 1478, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/git.ceph.com_ceph_f18f3f1cbd11e4a41c557e616d172f24deb91473/qa/tasks/ceph_manager.py", line 3204, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2024-06-18T03:03:34.040 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-18T03:03:34.069 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-18T03:03:34.101 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-06-18T03:03:34.101 DEBUG:teuthology.orchestra.run.smithi016:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-06-18T03:03:34.125 DEBUG:teuthology.orchestra.run.smithi123:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-06-18T03:03:34.158 INFO:tasks.cephadm:Stopping all daemons... 2024-06-18T03:03:34.159 INFO:tasks.cephadm.mon.smithi016:Stopping mon.smithi016... 2024-06-18T03:03:34.159 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968@mon.smithi016 2024-06-18T03:03:34.297 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:03:34 smithi016 systemd[1]: Stopping Ceph mon.smithi016 for 9e93709a-2d1d-11ef-bc9d-c7b262605968... 2024-06-18T03:03:34.297 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:03:34 smithi016 ceph-mon[29214]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:03:34.297 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:03:34 smithi016 ceph-mon[29214]: from='client.? 172.21.15.16:0/2570885801' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:03:34.484 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:03:34 smithi123 ceph-mon[36358]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-18T03:03:34.484 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:03:34 smithi123 ceph-mon[36358]: from='client.? 172.21.15.16:0/2570885801' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-18T03:03:34.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:03:34 smithi016 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-mon-smithi016[29210]: 2024-06-18T03:03:34.465+0000 7f55eae4d640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi016 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-06-18T03:03:34.555 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:03:34 smithi016 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-mon-smithi016[29210]: 2024-06-18T03:03:34.465+0000 7f55eae4d640 -1 mon.smithi016@0(leader) e2 *** Got Signal Terminated *** 2024-06-18T03:03:34.883 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:03:34 smithi016 podman[80006]: 2024-06-18 03:03:34.588066854 +0000 UTC m=+0.218998656 container died 01683559b4831b09c45624ab3e81d26706028b453ca350541850739dd740d737 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-mon-smithi016, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=quincy-3fe4105, org.label-schema.build-date=20240612) 2024-06-18T03:03:35.175 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:03:34 smithi016 podman[80006]: 2024-06-18 03:03:34.974325406 +0000 UTC m=+0.605257208 container cleanup 01683559b4831b09c45624ab3e81d26706028b453ca350541850739dd740d737 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-mon-smithi016, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , RELEASE=quincy-3fe4105, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240612, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-06-18T03:03:35.176 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:03:34 smithi016 bash[80006]: ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-mon-smithi016 2024-06-18T03:03:35.427 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968@mon.smithi016.service' 2024-06-18T03:03:35.450 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:03:35 smithi016 podman[80085]: 2024-06-18 03:03:35.175776807 +0000 UTC m=+0.586473542 container remove 01683559b4831b09c45624ab3e81d26706028b453ca350541850739dd740d737 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-mon-smithi016, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240612, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, RELEASE=quincy-3fe4105) 2024-06-18T03:03:35.450 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:03:35 smithi016 systemd[1]: ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968@mon.smithi016.service: Deactivated successfully. 2024-06-18T03:03:35.451 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:03:35 smithi016 systemd[1]: Stopped Ceph mon.smithi016 for 9e93709a-2d1d-11ef-bc9d-c7b262605968. 2024-06-18T03:03:35.451 INFO:journalctl@ceph.mon.smithi016.smithi016.stdout:Jun 18 03:03:35 smithi016 systemd[1]: ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968@mon.smithi016.service: Consumed 9.088s CPU time. 2024-06-18T03:03:35.470 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-18T03:03:35.471 INFO:tasks.cephadm.mon.smithi016:Stopped mon.smithi016 2024-06-18T03:03:35.471 INFO:tasks.cephadm.mon.smithi123:Stopping mon.smithi123... 2024-06-18T03:03:35.471 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968@mon.smithi123 2024-06-18T03:03:35.761 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:03:35 smithi123 systemd[1]: Stopping Ceph mon.smithi123 for 9e93709a-2d1d-11ef-bc9d-c7b262605968... 2024-06-18T03:03:36.234 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:03:35 smithi123 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-mon-smithi123[36341]: 2024-06-18T03:03:35.760+0000 7fb399364640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi123 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-06-18T03:03:36.235 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:03:35 smithi123 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-mon-smithi123[36341]: 2024-06-18T03:03:35.760+0000 7fb399364640 -1 mon.smithi123@1(peon) e2 *** Got Signal Terminated *** 2024-06-18T03:03:36.235 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:03:35 smithi123 podman[60559]: 2024-06-18 03:03:35.945756846 +0000 UTC m=+0.272134568 container died 59acc9db717fe26b7a2eceb760fe87a3a16c4a1592f2c13d1749d0167e55c9f5 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-mon-smithi123, ceph=True, org.label-schema.vendor=CentOS, RELEASE=quincy-3fe4105, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240612, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0) 2024-06-18T03:03:36.679 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:03:36 smithi123 podman[60559]: 2024-06-18 03:03:36.428653525 +0000 UTC m=+0.755031246 container cleanup 59acc9db717fe26b7a2eceb760fe87a3a16c4a1592f2c13d1749d0167e55c9f5 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-mon-smithi123, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240612, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, RELEASE=quincy-3fe4105, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47) 2024-06-18T03:03:36.680 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Jun 18 03:03:36 smithi123 bash[60559]: ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-mon-smithi123 2024-06-18T03:03:36.816 DEBUG:teuthology.orchestra.run.smithi123:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968@mon.smithi123.service' 2024-06-18T03:03:36.861 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-18T03:03:36.862 INFO:tasks.cephadm.mon.smithi123:Stopped mon.smithi123 2024-06-18T03:03:36.862 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-06-18T03:03:36.862 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968@osd.0 2024-06-18T03:03:37.171 INFO:journalctl@ceph.osd.0.smithi016.stdout:Jun 18 03:03:36 smithi016 systemd[1]: Stopping Ceph osd.0 for 9e93709a-2d1d-11ef-bc9d-c7b262605968... 2024-06-18T03:03:37.555 INFO:journalctl@ceph.osd.0.smithi016.stdout:Jun 18 03:03:37 smithi016 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-0[47382]: 2024-06-18T03:03:37.170+0000 7f6e3ee43640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-18T03:03:37.555 INFO:journalctl@ceph.osd.0.smithi016.stdout:Jun 18 03:03:37 smithi016 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-0[47382]: 2024-06-18T03:03:37.170+0000 7f6e3ee43640 -1 osd.0 41 *** Got signal Terminated *** 2024-06-18T03:03:37.556 INFO:journalctl@ceph.osd.0.smithi016.stdout:Jun 18 03:03:37 smithi016 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-0[47382]: 2024-06-18T03:03:37.170+0000 7f6e3ee43640 -1 osd.0 41 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-18T03:03:42.938 INFO:journalctl@ceph.osd.0.smithi016.stdout:Jun 18 03:03:42 smithi016 podman[80225]: 2024-06-18 03:03:42.621200563 +0000 UTC m=+5.560981225 container died e570072095e6cbed27bb637aad3b707481c565e2760bb1b384c412d1ebdf63f1 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-0, org.label-schema.license=GPLv2, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, RELEASE=quincy-3fe4105, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240612, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-06-18T03:03:43.305 INFO:journalctl@ceph.osd.0.smithi016.stdout:Jun 18 03:03:43 smithi016 podman[80225]: 2024-06-18 03:03:43.137577844 +0000 UTC m=+6.077358506 container cleanup e570072095e6cbed27bb637aad3b707481c565e2760bb1b384c412d1ebdf63f1 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-0, RELEASE=quincy-3fe4105, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240612, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True) 2024-06-18T03:03:43.305 INFO:journalctl@ceph.osd.0.smithi016.stdout:Jun 18 03:03:43 smithi016 bash[80225]: ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-0 2024-06-18T03:03:43.784 INFO:journalctl@ceph.osd.0.smithi016.stdout:Jun 18 03:03:43 smithi016 podman[80241]: 2024-06-18 03:03:43.389615941 +0000 UTC m=+0.765016687 container remove e570072095e6cbed27bb637aad3b707481c565e2760bb1b384c412d1ebdf63f1 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-0, GIT_CLEAN=True, ceph=True, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240612, RELEASE=quincy-3fe4105, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, CEPH_POINT_RELEASE=) 2024-06-18T03:03:44.055 INFO:journalctl@ceph.osd.0.smithi016.stdout:Jun 18 03:03:43 smithi016 podman[80332]: 2024-06-18 03:03:43.840560562 +0000 UTC m=+0.101758613 container create f238690d68978ef3ba11527b3581da80eeb00cee9f7b8729c8d6e30a2d7be161 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, RELEASE=quincy-3fe4105, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.build-date=20240612, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-06-18T03:03:44.055 INFO:journalctl@ceph.osd.0.smithi016.stdout:Jun 18 03:03:43 smithi016 podman[80332]: 2024-06-18 03:03:43.756881239 +0000 UTC m=+0.018079292 image pull 1a271607156e1b6becab587233eefa647399ff75aa2bb76fdb2f0360ec5b7818 quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032 2024-06-18T03:03:44.370 INFO:journalctl@ceph.osd.0.smithi016.stdout:Jun 18 03:03:44 smithi016 podman[80332]: 2024-06-18 03:03:44.105208621 +0000 UTC m=+0.366406676 container init f238690d68978ef3ba11527b3581da80eeb00cee9f7b8729c8d6e30a2d7be161 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=quincy-3fe4105, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.build-date=20240612, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-06-18T03:03:44.370 INFO:journalctl@ceph.osd.0.smithi016.stdout:Jun 18 03:03:44 smithi016 podman[80332]: 2024-06-18 03:03:44.108689884 +0000 UTC m=+0.369887937 container start f238690d68978ef3ba11527b3581da80eeb00cee9f7b8729c8d6e30a2d7be161 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240612, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=quincy-3fe4105, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-18T03:03:44.370 INFO:journalctl@ceph.osd.0.smithi016.stdout:Jun 18 03:03:44 smithi016 podman[80332]: 2024-06-18 03:03:44.209925929 +0000 UTC m=+0.471123986 container attach f238690d68978ef3ba11527b3581da80eeb00cee9f7b8729c8d6e30a2d7be161 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-0-deactivate, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240612, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=quincy-3fe4105, org.label-schema.vendor=CentOS) 2024-06-18T03:03:44.806 INFO:journalctl@ceph.osd.0.smithi016.stdout:Jun 18 03:03:44 smithi016 podman[80332]: 2024-06-18 03:03:44.370556168 +0000 UTC m=+0.631754221 container died f238690d68978ef3ba11527b3581da80eeb00cee9f7b8729c8d6e30a2d7be161 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, RELEASE=quincy-3fe4105, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240612, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-06-18T03:03:44.991 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968@osd.0.service' 2024-06-18T03:03:45.033 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-18T03:03:45.033 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-06-18T03:03:45.033 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-06-18T03:03:45.034 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968@osd.1 2024-06-18T03:03:45.401 INFO:journalctl@ceph.osd.1.smithi016.stdout:Jun 18 03:03:45 smithi016 systemd[1]: Stopping Ceph osd.1 for 9e93709a-2d1d-11ef-bc9d-c7b262605968... 2024-06-18T03:03:45.805 INFO:journalctl@ceph.osd.1.smithi016.stdout:Jun 18 03:03:45 smithi016 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-1[54640]: 2024-06-18T03:03:45.400+0000 7f3a6c6ba640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-18T03:03:45.805 INFO:journalctl@ceph.osd.1.smithi016.stdout:Jun 18 03:03:45 smithi016 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-1[54640]: 2024-06-18T03:03:45.400+0000 7f3a6c6ba640 -1 osd.1 41 *** Got signal Terminated *** 2024-06-18T03:03:45.805 INFO:journalctl@ceph.osd.1.smithi016.stdout:Jun 18 03:03:45 smithi016 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-1[54640]: 2024-06-18T03:03:45.400+0000 7f3a6c6ba640 -1 osd.1 41 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-18T03:03:51.110 INFO:journalctl@ceph.osd.1.smithi016.stdout:Jun 18 03:03:50 smithi016 podman[80433]: 2024-06-18 03:03:50.805861723 +0000 UTC m=+5.545884519 container died 773a9b78a2cd75117ad8fa4af7c7ac379f3738072457c8d024c90f2e34cb5261 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-1, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240612, RELEASE=quincy-3fe4105, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-18T03:03:51.477 INFO:journalctl@ceph.osd.1.smithi016.stdout:Jun 18 03:03:51 smithi016 podman[80433]: 2024-06-18 03:03:51.275888842 +0000 UTC m=+6.015911635 container cleanup 773a9b78a2cd75117ad8fa4af7c7ac379f3738072457c8d024c90f2e34cb5261 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-1, RELEASE=quincy-3fe4105, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240612) 2024-06-18T03:03:51.477 INFO:journalctl@ceph.osd.1.smithi016.stdout:Jun 18 03:03:51 smithi016 bash[80433]: ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-1 2024-06-18T03:03:51.805 INFO:journalctl@ceph.osd.1.smithi016.stdout:Jun 18 03:03:51 smithi016 podman[80450]: 2024-06-18 03:03:51.477451036 +0000 UTC m=+0.663414508 container remove 773a9b78a2cd75117ad8fa4af7c7ac379f3738072457c8d024c90f2e34cb5261 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-1, ceph=True, org.label-schema.build-date=20240612, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=quincy-3fe4105, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0) 2024-06-18T03:03:52.197 INFO:journalctl@ceph.osd.1.smithi016.stdout:Jun 18 03:03:51 smithi016 podman[80538]: 2024-06-18 03:03:51.903330961 +0000 UTC m=+0.111655796 container create c4e91a847ddbb142832dd3e0f1e78bbb1a65f59c5c814e561c6103ddb4347a7c (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-1-deactivate, RELEASE=quincy-3fe4105, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240612, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-06-18T03:03:52.197 INFO:journalctl@ceph.osd.1.smithi016.stdout:Jun 18 03:03:51 smithi016 podman[80538]: 2024-06-18 03:03:51.80989356 +0000 UTC m=+0.018218411 image pull 1a271607156e1b6becab587233eefa647399ff75aa2bb76fdb2f0360ec5b7818 quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032 2024-06-18T03:03:52.198 INFO:journalctl@ceph.osd.1.smithi016.stdout:Jun 18 03:03:52 smithi016 podman[80538]: 2024-06-18 03:03:52.122823378 +0000 UTC m=+0.331148226 container init c4e91a847ddbb142832dd3e0f1e78bbb1a65f59c5c814e561c6103ddb4347a7c (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-1-deactivate, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, RELEASE=quincy-3fe4105, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240612) 2024-06-18T03:03:52.198 INFO:journalctl@ceph.osd.1.smithi016.stdout:Jun 18 03:03:52 smithi016 podman[80538]: 2024-06-18 03:03:52.127436422 +0000 UTC m=+0.335761258 container start c4e91a847ddbb142832dd3e0f1e78bbb1a65f59c5c814e561c6103ddb4347a7c (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-1-deactivate, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.build-date=20240612, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=quincy-3fe4105, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True) 2024-06-18T03:03:52.198 INFO:journalctl@ceph.osd.1.smithi016.stdout:Jun 18 03:03:52 smithi016 podman[80538]: 2024-06-18 03:03:52.197779462 +0000 UTC m=+0.406104327 container attach c4e91a847ddbb142832dd3e0f1e78bbb1a65f59c5c814e561c6103ddb4347a7c (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-1-deactivate, RELEASE=quincy-3fe4105, org.label-schema.build-date=20240612, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-06-18T03:03:52.555 INFO:journalctl@ceph.osd.1.smithi016.stdout:Jun 18 03:03:52 smithi016 podman[80538]: 2024-06-18 03:03:52.378738821 +0000 UTC m=+0.587063658 container died c4e91a847ddbb142832dd3e0f1e78bbb1a65f59c5c814e561c6103ddb4347a7c (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-1-deactivate, RELEASE=quincy-3fe4105, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.build-date=20240612, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-06-18T03:03:53.088 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968@osd.1.service' 2024-06-18T03:03:53.132 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-18T03:03:53.132 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-06-18T03:03:53.132 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-06-18T03:03:53.132 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968@osd.2 2024-06-18T03:03:53.488 INFO:journalctl@ceph.osd.2.smithi016.stdout:Jun 18 03:03:53 smithi016 systemd[1]: Stopping Ceph osd.2 for 9e93709a-2d1d-11ef-bc9d-c7b262605968... 2024-06-18T03:03:53.805 INFO:journalctl@ceph.osd.2.smithi016.stdout:Jun 18 03:03:53 smithi016 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-2[62337]: 2024-06-18T03:03:53.487+0000 7f1b7f742640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-18T03:03:53.805 INFO:journalctl@ceph.osd.2.smithi016.stdout:Jun 18 03:03:53 smithi016 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-2[62337]: 2024-06-18T03:03:53.487+0000 7f1b7f742640 -1 osd.2 41 *** Got signal Terminated *** 2024-06-18T03:03:53.805 INFO:journalctl@ceph.osd.2.smithi016.stdout:Jun 18 03:03:53 smithi016 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-2[62337]: 2024-06-18T03:03:53.487+0000 7f1b7f742640 -1 osd.2 41 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-18T03:03:59.264 INFO:journalctl@ceph.osd.2.smithi016.stdout:Jun 18 03:03:58 smithi016 podman[80640]: 2024-06-18 03:03:58.982360977 +0000 UTC m=+5.613104353 container died 62c1ad97745cfdec1e53f83fe7ef175a1bc297077f23410ba93e9fe2e5e7a411 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=quincy-3fe4105, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.build-date=20240612, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-18T03:03:59.555 INFO:journalctl@ceph.osd.2.smithi016.stdout:Jun 18 03:03:59 smithi016 podman[80640]: 2024-06-18 03:03:59.42172427 +0000 UTC m=+6.052467688 container cleanup 62c1ad97745cfdec1e53f83fe7ef175a1bc297077f23410ba93e9fe2e5e7a411 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240612, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=quincy-3fe4105) 2024-06-18T03:03:59.555 INFO:journalctl@ceph.osd.2.smithi016.stdout:Jun 18 03:03:59 smithi016 bash[80640]: ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-2 2024-06-18T03:04:00.048 INFO:journalctl@ceph.osd.2.smithi016.stdout:Jun 18 03:03:59 smithi016 podman[80659]: 2024-06-18 03:03:59.598108021 +0000 UTC m=+0.606354231 container remove 62c1ad97745cfdec1e53f83fe7ef175a1bc297077f23410ba93e9fe2e5e7a411 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, RELEASE=quincy-3fe4105, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240612, GIT_BRANCH=HEAD) 2024-06-18T03:04:00.305 INFO:journalctl@ceph.osd.2.smithi016.stdout:Jun 18 03:04:00 smithi016 podman[80746]: 2024-06-18 03:04:00.048872011 +0000 UTC m=+0.114001262 container create b3be506f90a89c98456c1183988e61996c350a68b1c229f5261a3f11dbb362f0 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-2-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240612, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-3fe4105, org.label-schema.schema-version=1.0) 2024-06-18T03:04:00.305 INFO:journalctl@ceph.osd.2.smithi016.stdout:Jun 18 03:04:00 smithi016 podman[80746]: 2024-06-18 03:03:59.952742946 +0000 UTC m=+0.017872197 image pull 1a271607156e1b6becab587233eefa647399ff75aa2bb76fdb2f0360ec5b7818 quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032 2024-06-18T03:04:00.305 INFO:journalctl@ceph.osd.2.smithi016.stdout:Jun 18 03:04:00 smithi016 podman[80746]: 2024-06-18 03:04:00.28465948 +0000 UTC m=+0.349788747 container init b3be506f90a89c98456c1183988e61996c350a68b1c229f5261a3f11dbb362f0 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-2-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240612, RELEASE=quincy-3fe4105) 2024-06-18T03:04:00.305 INFO:journalctl@ceph.osd.2.smithi016.stdout:Jun 18 03:04:00 smithi016 podman[80746]: 2024-06-18 03:04:00.288046999 +0000 UTC m=+0.353176255 container start b3be506f90a89c98456c1183988e61996c350a68b1c229f5261a3f11dbb362f0 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, RELEASE=quincy-3fe4105, org.label-schema.build-date=20240612, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=) 2024-06-18T03:04:00.806 INFO:journalctl@ceph.osd.2.smithi016.stdout:Jun 18 03:04:00 smithi016 podman[80746]: 2024-06-18 03:04:00.393176598 +0000 UTC m=+0.458305866 container attach b3be506f90a89c98456c1183988e61996c350a68b1c229f5261a3f11dbb362f0 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-2-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-3fe4105, org.label-schema.build-date=20240612, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, ceph=True) 2024-06-18T03:04:00.806 INFO:journalctl@ceph.osd.2.smithi016.stdout:Jun 18 03:04:00 smithi016 podman[80746]: 2024-06-18 03:04:00.542564283 +0000 UTC m=+0.607693535 container died b3be506f90a89c98456c1183988e61996c350a68b1c229f5261a3f11dbb362f0 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-2-deactivate, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, RELEASE=quincy-3fe4105, org.label-schema.license=GPLv2, org.label-schema.build-date=20240612, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-18T03:04:01.096 INFO:journalctl@ceph.osd.2.smithi016.stdout:Jun 18 03:04:01 smithi016 podman[80770]: 2024-06-18 03:04:01.071809468 +0000 UTC m=+0.519984719 container remove b3be506f90a89c98456c1183988e61996c350a68b1c229f5261a3f11dbb362f0 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-2-deactivate, ceph=True, org.label-schema.license=GPLv2, RELEASE=quincy-3fe4105, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.build-date=20240612, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-06-18T03:04:01.102 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968@osd.2.service' 2024-06-18T03:04:01.141 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-18T03:04:01.142 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-06-18T03:04:01.142 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-06-18T03:04:01.142 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968@osd.3 2024-06-18T03:04:01.464 INFO:journalctl@ceph.osd.3.smithi123.stdout:Jun 18 03:04:01 smithi123 systemd[1]: Stopping Ceph osd.3 for 9e93709a-2d1d-11ef-bc9d-c7b262605968... 2024-06-18T03:04:01.734 INFO:journalctl@ceph.osd.3.smithi123.stdout:Jun 18 03:04:01 smithi123 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-3[41826]: 2024-06-18T03:04:01.464+0000 7fd10d8df640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-18T03:04:01.735 INFO:journalctl@ceph.osd.3.smithi123.stdout:Jun 18 03:04:01 smithi123 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-3[41826]: 2024-06-18T03:04:01.464+0000 7fd10d8df640 -1 osd.3 41 *** Got signal Terminated *** 2024-06-18T03:04:01.735 INFO:journalctl@ceph.osd.3.smithi123.stdout:Jun 18 03:04:01 smithi123 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-3[41826]: 2024-06-18T03:04:01.464+0000 7fd10d8df640 -1 osd.3 41 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-18T03:04:05.484 INFO:journalctl@ceph.osd.3.smithi123.stdout:Jun 18 03:04:05 smithi123 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-3[41826]: 2024-06-18T03:04:05.106+0000 7fd10aef9640 -1 osd.3 41 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-06-18T03:03:41.540124+0000 front 2024-06-18T03:03:41.539890+0000 (oldest deadline 2024-06-18T03:04:04.439958+0000) 2024-06-18T03:04:06.484 INFO:journalctl@ceph.osd.3.smithi123.stdout:Jun 18 03:04:06 smithi123 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-3[41826]: 2024-06-18T03:04:06.131+0000 7fd10aef9640 -1 osd.3 41 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-06-18T03:03:41.540124+0000 front 2024-06-18T03:03:41.539890+0000 (oldest deadline 2024-06-18T03:04:04.439958+0000) 2024-06-18T03:04:06.485 INFO:journalctl@ceph.osd.4.smithi123.stdout:Jun 18 03:04:05 smithi123 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-4[48669]: 2024-06-18T03:04:05.989+0000 7fb820648640 -1 osd.4 41 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-06-18T03:03:39.579521+0000 front 2024-06-18T03:03:39.579387+0000 (oldest deadline 2024-06-18T03:04:05.479343+0000) 2024-06-18T03:04:06.485 INFO:journalctl@ceph.osd.5.smithi123.stdout:Jun 18 03:04:06 smithi123 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-5[55214]: 2024-06-18T03:04:06.140+0000 7f954873e640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-06-18T03:03:40.744674+0000 front 2024-06-18T03:03:40.744579+0000 (oldest deadline 2024-06-18T03:04:05.444443+0000) 2024-06-18T03:04:06.974 INFO:journalctl@ceph.osd.3.smithi123.stdout:Jun 18 03:04:06 smithi123 podman[60699]: 2024-06-18 03:04:06.797028159 +0000 UTC m=+5.442713908 container died 5b2f4d8a71981cb47f44d1c1582afe5d6a1d37aa1c9490e5939ab48e99895ec9 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-3, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240612, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=quincy-3fe4105, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True) 2024-06-18T03:04:07.234 INFO:journalctl@ceph.osd.4.smithi123.stdout:Jun 18 03:04:06 smithi123 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-4[48669]: 2024-06-18T03:04:06.972+0000 7fb820648640 -1 osd.4 41 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-06-18T03:03:39.579521+0000 front 2024-06-18T03:03:39.579387+0000 (oldest deadline 2024-06-18T03:04:05.479343+0000) 2024-06-18T03:04:07.235 INFO:journalctl@ceph.osd.5.smithi123.stdout:Jun 18 03:04:07 smithi123 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-5[55214]: 2024-06-18T03:04:07.132+0000 7f954873e640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-06-18T03:03:40.744674+0000 front 2024-06-18T03:03:40.744579+0000 (oldest deadline 2024-06-18T03:04:05.444443+0000) 2024-06-18T03:04:07.235 INFO:journalctl@ceph.osd.3.smithi123.stdout:Jun 18 03:04:07 smithi123 podman[60699]: 2024-06-18 03:04:07.147278401 +0000 UTC m=+5.792964153 container cleanup 5b2f4d8a71981cb47f44d1c1582afe5d6a1d37aa1c9490e5939ab48e99895ec9 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-3, org.label-schema.build-date=20240612, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, GIT_CLEAN=True, RELEASE=quincy-3fe4105, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD) 2024-06-18T03:04:07.235 INFO:journalctl@ceph.osd.3.smithi123.stdout:Jun 18 03:04:07 smithi123 bash[60699]: ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-3 2024-06-18T03:04:07.734 INFO:journalctl@ceph.osd.3.smithi123.stdout:Jun 18 03:04:07 smithi123 podman[60716]: 2024-06-18 03:04:07.331600451 +0000 UTC m=+0.529067979 container remove 5b2f4d8a71981cb47f44d1c1582afe5d6a1d37aa1c9490e5939ab48e99895ec9 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-3, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240612, RELEASE=quincy-3fe4105, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-18T03:04:08.034 INFO:journalctl@ceph.osd.3.smithi123.stdout:Jun 18 03:04:07 smithi123 podman[60806]: 2024-06-18 03:04:07.649972502 +0000 UTC m=+0.016751663 image pull 1a271607156e1b6becab587233eefa647399ff75aa2bb76fdb2f0360ec5b7818 quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032 2024-06-18T03:04:08.034 INFO:journalctl@ceph.osd.3.smithi123.stdout:Jun 18 03:04:07 smithi123 podman[60806]: 2024-06-18 03:04:07.757207518 +0000 UTC m=+0.123986679 container create 2a6816aca12093dd1b5976c6084f2dd8d83b2fc3f8847e346a4e8af565889564 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-3-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.build-date=20240612, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, org.label-schema.schema-version=1.0, RELEASE=quincy-3fe4105, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-06-18T03:04:08.034 INFO:journalctl@ceph.osd.3.smithi123.stdout:Jun 18 03:04:07 smithi123 podman[60806]: 2024-06-18 03:04:07.942750602 +0000 UTC m=+0.309529767 container init 2a6816aca12093dd1b5976c6084f2dd8d83b2fc3f8847e346a4e8af565889564 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240612, RELEASE=quincy-3fe4105, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-06-18T03:04:08.034 INFO:journalctl@ceph.osd.3.smithi123.stdout:Jun 18 03:04:07 smithi123 podman[60806]: 2024-06-18 03:04:07.946402136 +0000 UTC m=+0.313181298 container start 2a6816aca12093dd1b5976c6084f2dd8d83b2fc3f8847e346a4e8af565889564 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-3-deactivate, ceph=True, io.buildah.version=1.36.0, org.label-schema.build-date=20240612, RELEASE=quincy-3fe4105, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-06-18T03:04:08.035 INFO:journalctl@ceph.osd.3.smithi123.stdout:Jun 18 03:04:08 smithi123 podman[60806]: 2024-06-18 03:04:08.034317334 +0000 UTC m=+0.401096504 container attach 2a6816aca12093dd1b5976c6084f2dd8d83b2fc3f8847e346a4e8af565889564 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, RELEASE=quincy-3fe4105, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240612, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-06-18T03:04:08.035 INFO:journalctl@ceph.osd.4.smithi123.stdout:Jun 18 03:04:07 smithi123 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-4[48669]: 2024-06-18T03:04:07.974+0000 7fb820648640 -1 osd.4 41 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-06-18T03:03:39.579521+0000 front 2024-06-18T03:03:39.579387+0000 (oldest deadline 2024-06-18T03:04:05.479343+0000) 2024-06-18T03:04:08.457 INFO:journalctl@ceph.osd.5.smithi123.stdout:Jun 18 03:04:08 smithi123 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-5[55214]: 2024-06-18T03:04:08.135+0000 7f954873e640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-06-18T03:03:40.744674+0000 front 2024-06-18T03:03:40.744579+0000 (oldest deadline 2024-06-18T03:04:05.444443+0000) 2024-06-18T03:04:08.457 INFO:journalctl@ceph.osd.3.smithi123.stdout:Jun 18 03:04:08 smithi123 podman[60806]: 2024-06-18 03:04:08.19028399 +0000 UTC m=+0.557063152 container died 2a6816aca12093dd1b5976c6084f2dd8d83b2fc3f8847e346a4e8af565889564 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.36.0, org.label-schema.build-date=20240612, RELEASE=quincy-3fe4105, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS) 2024-06-18T03:04:08.775 DEBUG:teuthology.orchestra.run.smithi123:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968@osd.3.service' 2024-06-18T03:04:08.812 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-18T03:04:08.812 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-06-18T03:04:08.812 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-06-18T03:04:08.812 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968@osd.4 2024-06-18T03:04:09.152 INFO:journalctl@ceph.osd.4.smithi123.stdout:Jun 18 03:04:08 smithi123 systemd[1]: Stopping Ceph osd.4 for 9e93709a-2d1d-11ef-bc9d-c7b262605968... 2024-06-18T03:04:09.152 INFO:journalctl@ceph.osd.4.smithi123.stdout:Jun 18 03:04:09 smithi123 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-4[48669]: 2024-06-18T03:04:09.003+0000 7fb820648640 -1 osd.4 41 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-06-18T03:03:39.579521+0000 front 2024-06-18T03:03:39.579387+0000 (oldest deadline 2024-06-18T03:04:05.479343+0000) 2024-06-18T03:04:09.423 INFO:journalctl@ceph.osd.4.smithi123.stdout:Jun 18 03:04:09 smithi123 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-4[48669]: 2024-06-18T03:04:09.174+0000 7fb825032640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-18T03:04:09.423 INFO:journalctl@ceph.osd.4.smithi123.stdout:Jun 18 03:04:09 smithi123 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-4[48669]: 2024-06-18T03:04:09.174+0000 7fb825032640 -1 osd.4 41 *** Got signal Terminated *** 2024-06-18T03:04:09.423 INFO:journalctl@ceph.osd.4.smithi123.stdout:Jun 18 03:04:09 smithi123 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-4[48669]: 2024-06-18T03:04:09.174+0000 7fb825032640 -1 osd.4 41 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-18T03:04:09.423 INFO:journalctl@ceph.osd.5.smithi123.stdout:Jun 18 03:04:09 smithi123 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-5[55214]: 2024-06-18T03:04:09.151+0000 7f954873e640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-06-18T03:03:40.744674+0000 front 2024-06-18T03:03:40.744579+0000 (oldest deadline 2024-06-18T03:04:05.444443+0000) 2024-06-18T03:04:10.484 INFO:journalctl@ceph.osd.4.smithi123.stdout:Jun 18 03:04:10 smithi123 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-4[48669]: 2024-06-18T03:04:10.048+0000 7fb820648640 -1 osd.4 41 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-06-18T03:03:39.579521+0000 front 2024-06-18T03:03:39.579387+0000 (oldest deadline 2024-06-18T03:04:05.479343+0000) 2024-06-18T03:04:10.484 INFO:journalctl@ceph.osd.5.smithi123.stdout:Jun 18 03:04:10 smithi123 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-5[55214]: 2024-06-18T03:04:10.159+0000 7f954873e640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-06-18T03:03:40.744674+0000 front 2024-06-18T03:03:40.744579+0000 (oldest deadline 2024-06-18T03:04:05.444443+0000) 2024-06-18T03:04:11.484 INFO:journalctl@ceph.osd.4.smithi123.stdout:Jun 18 03:04:11 smithi123 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-4[48669]: 2024-06-18T03:04:11.030+0000 7fb820648640 -1 osd.4 41 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-06-18T03:03:39.579521+0000 front 2024-06-18T03:03:39.579387+0000 (oldest deadline 2024-06-18T03:04:05.479343+0000) 2024-06-18T03:04:11.485 INFO:journalctl@ceph.osd.5.smithi123.stdout:Jun 18 03:04:11 smithi123 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-5[55214]: 2024-06-18T03:04:11.117+0000 7f954873e640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-06-18T03:03:40.744674+0000 front 2024-06-18T03:03:40.744579+0000 (oldest deadline 2024-06-18T03:04:05.444443+0000) 2024-06-18T03:04:12.484 INFO:journalctl@ceph.osd.4.smithi123.stdout:Jun 18 03:04:12 smithi123 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-4[48669]: 2024-06-18T03:04:12.021+0000 7fb820648640 -1 osd.4 41 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-06-18T03:03:39.579521+0000 front 2024-06-18T03:03:39.579387+0000 (oldest deadline 2024-06-18T03:04:05.479343+0000) 2024-06-18T03:04:12.485 INFO:journalctl@ceph.osd.5.smithi123.stdout:Jun 18 03:04:12 smithi123 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-5[55214]: 2024-06-18T03:04:12.068+0000 7f954873e640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-06-18T03:03:40.744674+0000 front 2024-06-18T03:03:40.744579+0000 (oldest deadline 2024-06-18T03:04:05.444443+0000) 2024-06-18T03:04:12.485 INFO:journalctl@ceph.osd.5.smithi123.stdout:Jun 18 03:04:12 smithi123 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-5[55214]: 2024-06-18T03:04:12.068+0000 7f954873e640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-06-18T03:03:45.444943+0000 front 2024-06-18T03:03:45.444921+0000 (oldest deadline 2024-06-18T03:04:11.344711+0000) 2024-06-18T03:04:13.235 INFO:journalctl@ceph.osd.4.smithi123.stdout:Jun 18 03:04:12 smithi123 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-4[48669]: 2024-06-18T03:04:12.983+0000 7fb820648640 -1 osd.4 41 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-06-18T03:03:39.579521+0000 front 2024-06-18T03:03:39.579387+0000 (oldest deadline 2024-06-18T03:04:05.479343+0000) 2024-06-18T03:04:13.235 INFO:journalctl@ceph.osd.5.smithi123.stdout:Jun 18 03:04:13 smithi123 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-5[55214]: 2024-06-18T03:04:13.101+0000 7f954873e640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-06-18T03:03:40.744674+0000 front 2024-06-18T03:03:40.744579+0000 (oldest deadline 2024-06-18T03:04:05.444443+0000) 2024-06-18T03:04:13.235 INFO:journalctl@ceph.osd.5.smithi123.stdout:Jun 18 03:04:13 smithi123 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-5[55214]: 2024-06-18T03:04:13.101+0000 7f954873e640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-06-18T03:03:45.444943+0000 front 2024-06-18T03:03:45.444921+0000 (oldest deadline 2024-06-18T03:04:11.344711+0000) 2024-06-18T03:04:14.484 INFO:journalctl@ceph.osd.4.smithi123.stdout:Jun 18 03:04:14 smithi123 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-4[48669]: 2024-06-18T03:04:14.014+0000 7fb820648640 -1 osd.4 41 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-06-18T03:03:39.579521+0000 front 2024-06-18T03:03:39.579387+0000 (oldest deadline 2024-06-18T03:04:05.479343+0000) 2024-06-18T03:04:14.484 INFO:journalctl@ceph.osd.4.smithi123.stdout:Jun 18 03:04:14 smithi123 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-4[48669]: 2024-06-18T03:04:14.014+0000 7fb820648640 -1 osd.4 41 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-06-18T03:03:50.180230+0000 front 2024-06-18T03:03:50.180168+0000 (oldest deadline 2024-06-18T03:04:13.680000+0000) 2024-06-18T03:04:14.485 INFO:journalctl@ceph.osd.5.smithi123.stdout:Jun 18 03:04:14 smithi123 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-5[55214]: 2024-06-18T03:04:14.083+0000 7f954873e640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-06-18T03:03:40.744674+0000 front 2024-06-18T03:03:40.744579+0000 (oldest deadline 2024-06-18T03:04:05.444443+0000) 2024-06-18T03:04:14.485 INFO:journalctl@ceph.osd.5.smithi123.stdout:Jun 18 03:04:14 smithi123 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-5[55214]: 2024-06-18T03:04:14.083+0000 7f954873e640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-06-18T03:03:45.444943+0000 front 2024-06-18T03:03:45.444921+0000 (oldest deadline 2024-06-18T03:04:11.344711+0000) 2024-06-18T03:04:14.907 INFO:journalctl@ceph.osd.4.smithi123.stdout:Jun 18 03:04:14 smithi123 podman[60919]: 2024-06-18 03:04:14.646846606 +0000 UTC m=+5.588024817 container died b02a617f5bc87398fd9bcc53ce026e6f5b173cc6747c2829440d9506f605d99d (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-4, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240612, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=quincy-3fe4105, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, GIT_BRANCH=HEAD, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47) 2024-06-18T03:04:15.234 INFO:journalctl@ceph.osd.5.smithi123.stdout:Jun 18 03:04:15 smithi123 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-5[55214]: 2024-06-18T03:04:15.128+0000 7f954873e640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-06-18T03:03:40.744674+0000 front 2024-06-18T03:03:40.744579+0000 (oldest deadline 2024-06-18T03:04:05.444443+0000) 2024-06-18T03:04:15.234 INFO:journalctl@ceph.osd.5.smithi123.stdout:Jun 18 03:04:15 smithi123 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-5[55214]: 2024-06-18T03:04:15.128+0000 7f954873e640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-06-18T03:03:45.444943+0000 front 2024-06-18T03:03:45.444921+0000 (oldest deadline 2024-06-18T03:04:11.344711+0000) 2024-06-18T03:04:15.235 INFO:journalctl@ceph.osd.4.smithi123.stdout:Jun 18 03:04:14 smithi123 podman[60919]: 2024-06-18 03:04:14.990710939 +0000 UTC m=+5.931889158 container cleanup b02a617f5bc87398fd9bcc53ce026e6f5b173cc6747c2829440d9506f605d99d (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-4, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=quincy-3fe4105, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240612, CEPH_POINT_RELEASE=) 2024-06-18T03:04:15.235 INFO:journalctl@ceph.osd.4.smithi123.stdout:Jun 18 03:04:14 smithi123 bash[60919]: ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-4 2024-06-18T03:04:15.235 INFO:journalctl@ceph.osd.4.smithi123.stdout:Jun 18 03:04:15 smithi123 podman[60935]: 2024-06-18 03:04:15.141827576 +0000 UTC m=+0.494648599 container remove b02a617f5bc87398fd9bcc53ce026e6f5b173cc6747c2829440d9506f605d99d (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-4, org.label-schema.build-date=20240612, ceph=True, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, CEPH_POINT_RELEASE=, RELEASE=quincy-3fe4105, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-18T03:04:15.869 INFO:journalctl@ceph.osd.4.smithi123.stdout:Jun 18 03:04:15 smithi123 podman[61022]: 2024-06-18 03:04:15.559095856 +0000 UTC m=+0.112135755 container create 221d93365c39eef77f17ff7eec79cde25a5f26101169a2116f4ad2cfc5c4c243 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-4-deactivate, org.label-schema.build-date=20240612, maintainer=Guillaume Abrioux , RELEASE=quincy-3fe4105, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47) 2024-06-18T03:04:15.869 INFO:journalctl@ceph.osd.4.smithi123.stdout:Jun 18 03:04:15 smithi123 podman[61022]: 2024-06-18 03:04:15.463591175 +0000 UTC m=+0.016631082 image pull 1a271607156e1b6becab587233eefa647399ff75aa2bb76fdb2f0360ec5b7818 quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032 2024-06-18T03:04:15.869 INFO:journalctl@ceph.osd.4.smithi123.stdout:Jun 18 03:04:15 smithi123 podman[61022]: 2024-06-18 03:04:15.778074575 +0000 UTC m=+0.331114480 container init 221d93365c39eef77f17ff7eec79cde25a5f26101169a2116f4ad2cfc5c4c243 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-4-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=quincy-3fe4105, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240612, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-18T03:04:15.869 INFO:journalctl@ceph.osd.4.smithi123.stdout:Jun 18 03:04:15 smithi123 podman[61022]: 2024-06-18 03:04:15.781315755 +0000 UTC m=+0.334355649 container start 221d93365c39eef77f17ff7eec79cde25a5f26101169a2116f4ad2cfc5c4c243 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-4-deactivate, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240612, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=quincy-3fe4105, ceph=True, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-06-18T03:04:15.869 INFO:journalctl@ceph.osd.4.smithi123.stdout:Jun 18 03:04:15 smithi123 podman[61022]: 2024-06-18 03:04:15.869811675 +0000 UTC m=+0.422851569 container attach 221d93365c39eef77f17ff7eec79cde25a5f26101169a2116f4ad2cfc5c4c243 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-4-deactivate, org.label-schema.build-date=20240612, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, RELEASE=quincy-3fe4105, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-06-18T03:04:16.126 INFO:journalctl@ceph.osd.4.smithi123.stdout:Jun 18 03:04:16 smithi123 podman[61022]: 2024-06-18 03:04:16.023021976 +0000 UTC m=+0.576061869 container died 221d93365c39eef77f17ff7eec79cde25a5f26101169a2116f4ad2cfc5c4c243 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-4-deactivate, RELEASE=quincy-3fe4105, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240612, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True) 2024-06-18T03:04:16.126 INFO:journalctl@ceph.osd.5.smithi123.stdout:Jun 18 03:04:16 smithi123 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-5[55214]: 2024-06-18T03:04:16.125+0000 7f954873e640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-06-18T03:03:40.744674+0000 front 2024-06-18T03:03:40.744579+0000 (oldest deadline 2024-06-18T03:04:05.444443+0000) 2024-06-18T03:04:16.127 INFO:journalctl@ceph.osd.5.smithi123.stdout:Jun 18 03:04:16 smithi123 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-5[55214]: 2024-06-18T03:04:16.125+0000 7f954873e640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-06-18T03:03:45.444943+0000 front 2024-06-18T03:03:45.444921+0000 (oldest deadline 2024-06-18T03:04:11.344711+0000) 2024-06-18T03:04:16.597 DEBUG:teuthology.orchestra.run.smithi123:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968@osd.4.service' 2024-06-18T03:04:16.636 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-18T03:04:16.637 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-06-18T03:04:16.637 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-06-18T03:04:16.637 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968@osd.5 2024-06-18T03:04:16.950 INFO:journalctl@ceph.osd.5.smithi123.stdout:Jun 18 03:04:16 smithi123 systemd[1]: Stopping Ceph osd.5 for 9e93709a-2d1d-11ef-bc9d-c7b262605968... 2024-06-18T03:04:17.234 INFO:journalctl@ceph.osd.5.smithi123.stdout:Jun 18 03:04:16 smithi123 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-5[55214]: 2024-06-18T03:04:16.949+0000 7f954f12c640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-18T03:04:17.234 INFO:journalctl@ceph.osd.5.smithi123.stdout:Jun 18 03:04:16 smithi123 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-5[55214]: 2024-06-18T03:04:16.949+0000 7f954f12c640 -1 osd.5 41 *** Got signal Terminated *** 2024-06-18T03:04:17.235 INFO:journalctl@ceph.osd.5.smithi123.stdout:Jun 18 03:04:16 smithi123 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-5[55214]: 2024-06-18T03:04:16.949+0000 7f954f12c640 -1 osd.5 41 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-18T03:04:17.235 INFO:journalctl@ceph.osd.5.smithi123.stdout:Jun 18 03:04:17 smithi123 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-5[55214]: 2024-06-18T03:04:17.160+0000 7f954873e640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-06-18T03:03:40.744674+0000 front 2024-06-18T03:03:40.744579+0000 (oldest deadline 2024-06-18T03:04:05.444443+0000) 2024-06-18T03:04:17.235 INFO:journalctl@ceph.osd.5.smithi123.stdout:Jun 18 03:04:17 smithi123 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-5[55214]: 2024-06-18T03:04:17.160+0000 7f954873e640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-06-18T03:03:45.444943+0000 front 2024-06-18T03:03:45.444921+0000 (oldest deadline 2024-06-18T03:04:11.344711+0000) 2024-06-18T03:04:18.484 INFO:journalctl@ceph.osd.5.smithi123.stdout:Jun 18 03:04:18 smithi123 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-5[55214]: 2024-06-18T03:04:18.173+0000 7f954873e640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-06-18T03:03:40.744674+0000 front 2024-06-18T03:03:40.744579+0000 (oldest deadline 2024-06-18T03:04:05.444443+0000) 2024-06-18T03:04:18.484 INFO:journalctl@ceph.osd.5.smithi123.stdout:Jun 18 03:04:18 smithi123 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-5[55214]: 2024-06-18T03:04:18.173+0000 7f954873e640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-06-18T03:03:45.444943+0000 front 2024-06-18T03:03:45.444921+0000 (oldest deadline 2024-06-18T03:04:11.344711+0000) 2024-06-18T03:04:19.484 INFO:journalctl@ceph.osd.5.smithi123.stdout:Jun 18 03:04:19 smithi123 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-5[55214]: 2024-06-18T03:04:19.213+0000 7f954873e640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-06-18T03:03:40.744674+0000 front 2024-06-18T03:03:40.744579+0000 (oldest deadline 2024-06-18T03:04:05.444443+0000) 2024-06-18T03:04:19.485 INFO:journalctl@ceph.osd.5.smithi123.stdout:Jun 18 03:04:19 smithi123 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-5[55214]: 2024-06-18T03:04:19.213+0000 7f954873e640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-06-18T03:03:45.444943+0000 front 2024-06-18T03:03:45.444921+0000 (oldest deadline 2024-06-18T03:04:11.344711+0000) 2024-06-18T03:04:20.484 INFO:journalctl@ceph.osd.5.smithi123.stdout:Jun 18 03:04:20 smithi123 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-5[55214]: 2024-06-18T03:04:20.200+0000 7f954873e640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-06-18T03:03:40.744674+0000 front 2024-06-18T03:03:40.744579+0000 (oldest deadline 2024-06-18T03:04:05.444443+0000) 2024-06-18T03:04:20.484 INFO:journalctl@ceph.osd.5.smithi123.stdout:Jun 18 03:04:20 smithi123 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-5[55214]: 2024-06-18T03:04:20.200+0000 7f954873e640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-06-18T03:03:45.444943+0000 front 2024-06-18T03:03:45.444921+0000 (oldest deadline 2024-06-18T03:04:11.344711+0000) 2024-06-18T03:04:20.485 INFO:journalctl@ceph.osd.5.smithi123.stdout:Jun 18 03:04:20 smithi123 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-5[55214]: 2024-06-18T03:04:20.200+0000 7f954873e640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.16:6822 osd.2 since back 2024-06-18T03:03:57.245531+0000 front 2024-06-18T03:03:57.245479+0000 (oldest deadline 2024-06-18T03:04:19.445664+0000) 2024-06-18T03:04:21.734 INFO:journalctl@ceph.osd.5.smithi123.stdout:Jun 18 03:04:21 smithi123 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-5[55214]: 2024-06-18T03:04:21.243+0000 7f954873e640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.16:6806 osd.0 since back 2024-06-18T03:03:40.744674+0000 front 2024-06-18T03:03:40.744579+0000 (oldest deadline 2024-06-18T03:04:05.444443+0000) 2024-06-18T03:04:21.735 INFO:journalctl@ceph.osd.5.smithi123.stdout:Jun 18 03:04:21 smithi123 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-5[55214]: 2024-06-18T03:04:21.243+0000 7f954873e640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.16:6814 osd.1 since back 2024-06-18T03:03:45.444943+0000 front 2024-06-18T03:03:45.444921+0000 (oldest deadline 2024-06-18T03:04:11.344711+0000) 2024-06-18T03:04:21.735 INFO:journalctl@ceph.osd.5.smithi123.stdout:Jun 18 03:04:21 smithi123 ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-5[55214]: 2024-06-18T03:04:21.243+0000 7f954873e640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.16:6822 osd.2 since back 2024-06-18T03:03:57.245531+0000 front 2024-06-18T03:03:57.245479+0000 (oldest deadline 2024-06-18T03:04:19.445664+0000) 2024-06-18T03:04:22.726 INFO:journalctl@ceph.osd.5.smithi123.stdout:Jun 18 03:04:22 smithi123 podman[61124]: 2024-06-18 03:04:22.463936731 +0000 UTC m=+5.595098240 container died 6d88f815ddf6a3059e641c6ea6d855932981662b04073eb2591336d835e60a15 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, RELEASE=quincy-3fe4105, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240612) 2024-06-18T03:04:22.984 INFO:journalctl@ceph.osd.5.smithi123.stdout:Jun 18 03:04:22 smithi123 podman[61124]: 2024-06-18 03:04:22.79997696 +0000 UTC m=+5.931138465 container cleanup 6d88f815ddf6a3059e641c6ea6d855932981662b04073eb2591336d835e60a15 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-5, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, CEPH_POINT_RELEASE=, RELEASE=quincy-3fe4105, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240612, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-06-18T03:04:22.984 INFO:journalctl@ceph.osd.5.smithi123.stdout:Jun 18 03:04:22 smithi123 bash[61124]: ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-5 2024-06-18T03:04:23.234 INFO:journalctl@ceph.osd.5.smithi123.stdout:Jun 18 03:04:22 smithi123 podman[61141]: 2024-06-18 03:04:22.984330814 +0000 UTC m=+0.516745984 container remove 6d88f815ddf6a3059e641c6ea6d855932981662b04073eb2591336d835e60a15 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-5, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-3fe4105, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240612, GIT_BRANCH=HEAD) 2024-06-18T03:04:23.687 INFO:journalctl@ceph.osd.5.smithi123.stdout:Jun 18 03:04:23 smithi123 podman[61230]: 2024-06-18 03:04:23.393434341 +0000 UTC m=+0.110807739 container create 7cfc887d9fe7600b880965a88de50dffaf19e410d79d4ad028a8bdf920fc18d7 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-5-deactivate, RELEASE=quincy-3fe4105, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.build-date=20240612, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-06-18T03:04:23.687 INFO:journalctl@ceph.osd.5.smithi123.stdout:Jun 18 03:04:23 smithi123 podman[61230]: 2024-06-18 03:04:23.299529129 +0000 UTC m=+0.016902533 image pull 1a271607156e1b6becab587233eefa647399ff75aa2bb76fdb2f0360ec5b7818 quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032 2024-06-18T03:04:23.687 INFO:journalctl@ceph.osd.5.smithi123.stdout:Jun 18 03:04:23 smithi123 podman[61230]: 2024-06-18 03:04:23.60418454 +0000 UTC m=+0.321557947 container init 7cfc887d9fe7600b880965a88de50dffaf19e410d79d4ad028a8bdf920fc18d7 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240612, org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=quincy-3fe4105, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=) 2024-06-18T03:04:23.687 INFO:journalctl@ceph.osd.5.smithi123.stdout:Jun 18 03:04:23 smithi123 podman[61230]: 2024-06-18 03:04:23.607459509 +0000 UTC m=+0.324832908 container start 7cfc887d9fe7600b880965a88de50dffaf19e410d79d4ad028a8bdf920fc18d7 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-5-deactivate, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240612, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-3fe4105, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-06-18T03:04:23.687 INFO:journalctl@ceph.osd.5.smithi123.stdout:Jun 18 03:04:23 smithi123 podman[61230]: 2024-06-18 03:04:23.687430722 +0000 UTC m=+0.404804149 container attach 7cfc887d9fe7600b880965a88de50dffaf19e410d79d4ad028a8bdf920fc18d7 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-5-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240612, ceph=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=quincy-3fe4105) 2024-06-18T03:04:23.984 INFO:journalctl@ceph.osd.5.smithi123.stdout:Jun 18 03:04:23 smithi123 podman[61230]: 2024-06-18 03:04:23.854418771 +0000 UTC m=+0.571792170 container died 7cfc887d9fe7600b880965a88de50dffaf19e410d79d4ad028a8bdf920fc18d7 (image=quay.ceph.io/ceph-ci/ceph@sha256:44bfca165b6fde12a114c44d7301e4ac7f01cd43bd80c4f282df050d5f79f032, name=ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968-osd-5-deactivate, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240612, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, RELEASE=quincy-3fe4105, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-06-18T03:04:24.424 DEBUG:teuthology.orchestra.run.smithi123:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9e93709a-2d1d-11ef-bc9d-c7b262605968@osd.5.service' 2024-06-18T03:04:24.462 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-18T03:04:24.462 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-06-18T03:04:24.462 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 --force --keep-logs 2024-06-18T03:04:33.794 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 --force --keep-logs 2024-06-18T03:04:38.680 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-18T03:04:38.714 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-18T03:04:38.740 INFO:tasks.cephadm:Archiving crash dumps... 2024-06-18T03:04:38.742 DEBUG:teuthology.misc:Transferring archived files from smithi016:/var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-06-17_21:08:03-orch-squid-distro-default-smithi/7760463/remote/smithi016/crash 2024-06-18T03:04:38.743 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar c -f - -C /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/crash -- . 2024-06-18T03:04:38.796 INFO:teuthology.orchestra.run.smithi016.stderr:tar: /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/crash: Cannot open: No such file or directory 2024-06-18T03:04:38.796 INFO:teuthology.orchestra.run.smithi016.stderr:tar: Error is not recoverable: exiting now 2024-06-18T03:04:38.798 DEBUG:teuthology.misc:Transferring archived files from smithi123:/var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-06-17_21:08:03-orch-squid-distro-default-smithi/7760463/remote/smithi123/crash 2024-06-18T03:04:38.799 DEBUG:teuthology.orchestra.run.smithi123:> sudo tar c -f - -C /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/crash -- . 2024-06-18T03:04:38.835 INFO:teuthology.orchestra.run.smithi123.stderr:tar: /var/lib/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/crash: Cannot open: No such file or directory 2024-06-18T03:04:38.835 INFO:teuthology.orchestra.run.smithi123.stderr:tar: Error is not recoverable: exiting now 2024-06-18T03:04:38.836 INFO:tasks.cephadm:Checking cluster log for badness... 2024-06-18T03:04:38.836 DEBUG:teuthology.orchestra.run.smithi016:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v OSD_DOWN | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | head -n 1 2024-06-18T03:04:38.878 INFO:teuthology.orchestra.run.smithi016.stdout:1718679199.2049313 mon.smithi016 (mon.0) 119 : cluster [WRN] Health check failed: 2 mgr modules have recently crashed (RECENT_MGR_MODULE_CRASH) 2024-06-18T03:04:38.878 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-06-18T03:04:38.879 DEBUG:teuthology.orchestra.run.smithi016:> sudo egrep '\[SEC\]' /var/log/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v OSD_DOWN | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | head -n 1 2024-06-18T03:04:38.954 DEBUG:teuthology.orchestra.run.smithi016:> sudo egrep '\[ERR\]' /var/log/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v OSD_DOWN | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | head -n 1 2024-06-18T03:04:39.029 INFO:teuthology.orchestra.run.smithi016.stdout:1718679275.9375453 mgr.smithi123.olpkyi (mgr.14236) 1 : cluster [ERR] Failed to load ceph-mgr modules: k8sevents 2024-06-18T03:04:39.030 INFO:tasks.cephadm:Compressing logs... 2024-06-18T03:04:39.030 DEBUG:teuthology.orchestra.run.smithi016:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-06-18T03:04:39.072 DEBUG:teuthology.orchestra.run.smithi123:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-06-18T03:04:39.098 INFO:teuthology.orchestra.run.smithi016.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-06-18T03:04:39.098 INFO:teuthology.orchestra.run.smithi016.stderr:: No such file or directory 2024-06-18T03:04:39.098 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/ceph.log 2024-06-18T03:04:39.098 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/ceph-osd.2.log 2024-06-18T03:04:39.098 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/ceph.audit.log 2024-06-18T03:04:39.098 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/ceph-mon.smithi016.log 2024-06-18T03:04:39.098 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/ceph-mgr.smithi016.amxydz.log 2024-06-18T03:04:39.099 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/ceph.audit.log: /var/log/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/ceph-mon.smithi016.log: gzip -5 --verbose -- /var/log/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/ceph-osd.1.log 2024-06-18T03:04:39.099 INFO:teuthology.orchestra.run.smithi016.stderr: 87.4% -- replaced with /var/log/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/ceph.log.gz 2024-06-18T03:04:39.099 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/ceph-mgr.smithi016.amxydz.log: gzip -5 --verbose -- /var/log/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/ceph-osd.0.log 2024-06-18T03:04:39.103 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/ceph.cephadm.log 2024-06-18T03:04:39.103 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/ceph-volume.log 2024-06-18T03:04:39.103 INFO:teuthology.orchestra.run.smithi016.stderr: 90.0%/var/log/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/ceph.cephadm.log: -- replaced with /var/log/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/ceph.audit.log.gz 2024-06-18T03:04:39.103 INFO:teuthology.orchestra.run.smithi016.stderr: 83.4% -- replaced with /var/log/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/ceph.cephadm.log.gz 2024-06-18T03:04:39.104 INFO:teuthology.orchestra.run.smithi123.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-06-18T03:04:39.104 INFO:teuthology.orchestra.run.smithi123.stderr:: No such file or directory 2024-06-18T03:04:39.104 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/ceph.log 2024-06-18T03:04:39.104 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/ceph.audit.log 2024-06-18T03:04:39.104 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/ceph-mgr.smithi123.olpkyi.log 2024-06-18T03:04:39.104 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/ceph-osd.5.log 2024-06-18T03:04:39.104 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose --/var/log/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/ceph-mgr.smithi123.olpkyi.log: /var/log/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/ceph-osd.3.log 2024-06-18T03:04:39.105 INFO:teuthology.orchestra.run.smithi123.stderr: 92.7% -- replaced with /var/log/ceph/cephadm.log.gz 2024-06-18T03:04:39.105 INFO:teuthology.orchestra.run.smithi123.stderr: 87.8% -- replaced with /var/log/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/ceph.log.gz 2024-06-18T03:04:39.105 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/ceph-osd.4.log 2024-06-18T03:04:39.105 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/ceph-mon.smithi123.log 2024-06-18T03:04:39.105 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/ceph.cephadm.log 2024-06-18T03:04:39.105 INFO:teuthology.orchestra.run.smithi123.stderr: 90.3% -- replaced with /var/log/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/ceph.audit.log.gz 2024-06-18T03:04:39.105 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/ceph-mon.smithi123.log: gzip -5 --verbose -- /var/log/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/ceph-volume.log 2024-06-18T03:04:39.105 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/ceph.cephadm.log: 81.8% -- replaced with /var/log/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/ceph.cephadm.log.gz 2024-06-18T03:04:39.105 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/ceph-volume.log: 92.4% -- replaced with /var/log/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/ceph-mgr.smithi123.olpkyi.log.gz 2024-06-18T03:04:39.105 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/ceph-volume.log: 93.7% -- replaced with /var/log/ceph/cephadm.log.gz 2024-06-18T03:04:39.108 INFO:teuthology.orchestra.run.smithi123.stderr: 90.9% -- replaced with /var/log/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/ceph-volume.log.gz 2024-06-18T03:04:39.111 INFO:teuthology.orchestra.run.smithi016.stderr: 90.9% -- replaced with /var/log/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/ceph-volume.log.gz 2024-06-18T03:04:39.144 INFO:teuthology.orchestra.run.smithi123.stderr: 92.9% -- replaced with /var/log/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/ceph-osd.4.log.gz 2024-06-18T03:04:39.152 INFO:teuthology.orchestra.run.smithi123.stderr: 93.0% -- replaced with /var/log/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/ceph-osd.5.log.gz 2024-06-18T03:04:39.155 INFO:teuthology.orchestra.run.smithi123.stderr: 92.4% -- replaced with /var/log/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/ceph-osd.3.log.gz 2024-06-18T03:04:39.155 INFO:teuthology.orchestra.run.smithi016.stderr: 93.0% -- replaced with /var/log/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/ceph-osd.2.log.gz 2024-06-18T03:04:39.164 INFO:teuthology.orchestra.run.smithi016.stderr: 92.8% -- replaced with /var/log/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/ceph-osd.0.log.gz 2024-06-18T03:04:39.189 INFO:teuthology.orchestra.run.smithi016.stderr: 93.2% -- replaced with /var/log/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/ceph-osd.1.log.gz 2024-06-18T03:04:39.192 INFO:teuthology.orchestra.run.smithi123.stderr: 92.9% -- replaced with /var/log/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/ceph-mon.smithi123.log.gz 2024-06-18T03:04:39.194 INFO:teuthology.orchestra.run.smithi123.stderr: 2024-06-18T03:04:39.194 INFO:teuthology.orchestra.run.smithi123.stderr:real 0m0.106s 2024-06-18T03:04:39.194 INFO:teuthology.orchestra.run.smithi123.stderr:user 0m0.266s 2024-06-18T03:04:39.194 INFO:teuthology.orchestra.run.smithi123.stderr:sys 0m0.025s 2024-06-18T03:04:39.230 INFO:teuthology.orchestra.run.smithi016.stderr: 89.4% -- replaced with /var/log/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/ceph-mgr.smithi016.amxydz.log.gz 2024-06-18T03:04:39.412 INFO:teuthology.orchestra.run.smithi016.stderr: 90.5% -- replaced with /var/log/ceph/9e93709a-2d1d-11ef-bc9d-c7b262605968/ceph-mon.smithi016.log.gz 2024-06-18T03:04:39.414 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-06-18T03:04:39.414 INFO:teuthology.orchestra.run.smithi016.stderr:real 0m0.328s 2024-06-18T03:04:39.414 INFO:teuthology.orchestra.run.smithi016.stderr:user 0m0.664s 2024-06-18T03:04:39.414 INFO:teuthology.orchestra.run.smithi016.stderr:sys 0m0.048s 2024-06-18T03:04:39.414 INFO:tasks.cephadm:Archiving logs... 2024-06-18T03:04:39.415 DEBUG:teuthology.misc:Transferring archived files from smithi016:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-06-17_21:08:03-orch-squid-distro-default-smithi/7760463/remote/smithi016/log 2024-06-18T03:04:39.416 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar c -f - -C /var/log/ceph -- . 2024-06-18T03:04:39.592 DEBUG:teuthology.misc:Transferring archived files from smithi123:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-06-17_21:08:03-orch-squid-distro-default-smithi/7760463/remote/smithi123/log 2024-06-18T03:04:39.593 DEBUG:teuthology.orchestra.run.smithi123:> sudo tar c -f - -C /var/log/ceph -- . 2024-06-18T03:04:39.667 INFO:tasks.cephadm:Removing cluster... 2024-06-18T03:04:39.667 DEBUG:teuthology.orchestra.run.smithi016:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 --force 2024-06-18T03:04:40.155 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 9e93709a-2d1d-11ef-bc9d-c7b262605968 --force 2024-06-18T03:04:40.618 INFO:tasks.cephadm:Removing cephadm ... 2024-06-18T03:04:40.618 DEBUG:teuthology.orchestra.run.smithi016:> rm -rf /home/ubuntu/cephtest/cephadm 2024-06-18T03:04:40.635 DEBUG:teuthology.orchestra.run.smithi123:> rm -rf /home/ubuntu/cephtest/cephadm 2024-06-18T03:04:40.651 INFO:tasks.cephadm:Teardown complete 2024-06-18T03:04:40.652 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8e9714173de9e92c97e8ef1045d333e96b793454/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_f18f3f1cbd11e4a41c557e616d172f24deb91473/qa/tasks/cephadm.py", line 2237, in task healthy(ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_ceph_f18f3f1cbd11e4a41c557e616d172f24deb91473/qa/tasks/ceph.py", line 1478, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/git.ceph.com_ceph_f18f3f1cbd11e4a41c557e616d172f24deb91473/qa/tasks/ceph_manager.py", line 3204, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2024-06-18T03:04:40.829 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=cdd56726f7434f7b917174f03ae409db Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8e9714173de9e92c97e8ef1045d333e96b793454/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_f18f3f1cbd11e4a41c557e616d172f24deb91473/qa/tasks/cephadm.py", line 2237, in task healthy(ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_ceph_f18f3f1cbd11e4a41c557e616d172f24deb91473/qa/tasks/ceph.py", line 1478, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/git.ceph.com_ceph_f18f3f1cbd11e4a41c557e616d172f24deb91473/qa/tasks/ceph_manager.py", line 3204, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2024-06-18T03:04:40.831 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-06-18T03:04:40.881 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-06-18T03:04:40.889 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-06-18T03:04:40.889 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-06-18T03:04:40.902 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-06-18T03:04:40.973 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-06-18T03:04:40.985 INFO:teuthology.task.clock:Checking final clock skew... 2024-06-18T03:04:40.986 DEBUG:teuthology.orchestra.run.smithi016:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-18T03:04:40.988 DEBUG:teuthology.orchestra.run.smithi123:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-18T03:04:41.002 INFO:teuthology.orchestra.run.smithi123.stderr:bash: line 1: ntpq: command not found 2024-06-18T03:04:41.003 INFO:teuthology.orchestra.run.smithi016.stderr:bash: line 1: ntpq: command not found 2024-06-18T03:04:41.006 INFO:teuthology.orchestra.run.smithi123.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-18T03:04:41.007 INFO:teuthology.orchestra.run.smithi123.stdout:=============================================================================== 2024-06-18T03:04:41.007 INFO:teuthology.orchestra.run.smithi123.stdout:^* hv01.front.sepia.ceph.com 3 6 377 38 +461us[ +480us] +/- 59ms 2024-06-18T03:04:41.007 INFO:teuthology.orchestra.run.smithi123.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 39 +239us[ +258us] +/- 76ms 2024-06-18T03:04:41.007 INFO:teuthology.orchestra.run.smithi123.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 39 -773us[ -755us] +/- 71ms 2024-06-18T03:04:41.007 INFO:teuthology.orchestra.run.smithi123.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-06-18T03:04:41.009 INFO:teuthology.orchestra.run.smithi016.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-18T03:04:41.009 INFO:teuthology.orchestra.run.smithi016.stdout:=============================================================================== 2024-06-18T03:04:41.009 INFO:teuthology.orchestra.run.smithi016.stdout:^* hv01.front.sepia.ceph.com 3 6 377 42 +412us[ +402us] +/- 58ms 2024-06-18T03:04:41.009 INFO:teuthology.orchestra.run.smithi016.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 38 +215us[ +215us] +/- 76ms 2024-06-18T03:04:41.009 INFO:teuthology.orchestra.run.smithi016.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 37 -815us[ -815us] +/- 71ms 2024-06-18T03:04:41.009 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-06-18T03:04:41.010 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-06-18T03:04:41.018 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-06-18T03:04:41.018 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-06-18T03:04:41.028 DEBUG:teuthology.orchestra.run.smithi016:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-06-18T03:04:41.127 DEBUG:teuthology.orchestra.run.smithi123:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-06-18T03:04:41.192 DEBUG:teuthology.orchestra.run.smithi016:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-18T03:04:41.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-18T03:04:41.223 DEBUG:teuthology.orchestra.run.smithi123:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-18T03:04:41.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-18T03:04:41.251 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-06-18T03:04:41.259 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-06-18T03:04:41.269 INFO:teuthology.task.internal:Duration was 1218.668236 seconds 2024-06-18T03:04:41.270 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-06-18T03:04:41.281 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-06-18T03:04:41.282 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-06-18T03:04:41.284 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-06-18T03:04:41.324 INFO:teuthology.orchestra.run.smithi016.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-18T03:04:41.329 INFO:teuthology.orchestra.run.smithi123.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-18T03:04:41.774 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-06-18T03:04:41.775 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi016.front.sepia.ceph.com 2024-06-18T03:04:41.775 DEBUG:teuthology.orchestra.run.smithi016:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-06-18T03:04:41.798 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi123.front.sepia.ceph.com 2024-06-18T03:04:41.798 DEBUG:teuthology.orchestra.run.smithi123:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-06-18T03:04:41.841 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-06-18T03:04:41.842 DEBUG:teuthology.orchestra.run.smithi016:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-06-18T03:04:41.844 DEBUG:teuthology.orchestra.run.smithi123:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-06-18T03:04:41.908 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-06-18T03:04:41.908 DEBUG:teuthology.orchestra.run.smithi016:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-06-18T03:04:41.912 DEBUG:teuthology.orchestra.run.smithi123:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-06-18T03:04:42.302 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-06-18T03:04:42.312 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-06-18T03:04:42.312 DEBUG:teuthology.orchestra.run.smithi016:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-06-18T03:04:42.337 DEBUG:teuthology.orchestra.run.smithi123:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-06-18T03:04:42.363 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-06-18T03:04:42.376 DEBUG:teuthology.orchestra.run.smithi016:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-06-18T03:04:42.379 DEBUG:teuthology.orchestra.run.smithi123:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-06-18T03:04:42.401 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern = core 2024-06-18T03:04:42.426 INFO:teuthology.orchestra.run.smithi123.stdout:kernel.core_pattern = core 2024-06-18T03:04:42.447 DEBUG:teuthology.orchestra.run.smithi016:> test -e /home/ubuntu/cephtest/archive/coredump 2024-06-18T03:04:42.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-18T03:04:42.482 DEBUG:teuthology.orchestra.run.smithi123:> test -e /home/ubuntu/cephtest/archive/coredump 2024-06-18T03:04:42.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-18T03:04:42.502 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-06-18T03:04:42.512 INFO:teuthology.task.internal:Transferring archived files... 2024-06-18T03:04:42.513 DEBUG:teuthology.misc:Transferring archived files from smithi016:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-06-17_21:08:03-orch-squid-distro-default-smithi/7760463/remote/smithi016 2024-06-18T03:04:42.513 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-06-18T03:04:42.563 DEBUG:teuthology.misc:Transferring archived files from smithi123:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-06-17_21:08:03-orch-squid-distro-default-smithi/7760463/remote/smithi123 2024-06-18T03:04:42.563 DEBUG:teuthology.orchestra.run.smithi123:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-06-18T03:04:42.599 INFO:teuthology.task.internal:Removing archive directory... 2024-06-18T03:04:42.600 DEBUG:teuthology.orchestra.run.smithi016:> rm -rf -- /home/ubuntu/cephtest/archive 2024-06-18T03:04:42.602 DEBUG:teuthology.orchestra.run.smithi123:> rm -rf -- /home/ubuntu/cephtest/archive 2024-06-18T03:04:42.648 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-06-18T03:04:42.659 INFO:teuthology.task.internal:Not uploading archives. 2024-06-18T03:04:42.659 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-06-18T03:04:42.669 INFO:teuthology.task.internal:Tidying up after the test... 2024-06-18T03:04:42.669 DEBUG:teuthology.orchestra.run.smithi016:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-06-18T03:04:42.671 DEBUG:teuthology.orchestra.run.smithi123:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-06-18T03:04:42.685 INFO:teuthology.orchestra.run.smithi016.stdout: 262165 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 18 03:04 /home/ubuntu/cephtest 2024-06-18T03:04:42.703 INFO:teuthology.orchestra.run.smithi123.stdout: 265695 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 18 03:04 /home/ubuntu/cephtest 2024-06-18T03:04:42.704 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-06-18T03:04:42.716 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-06-18T03:04:42.742 INFO:teuthology.run:Summary data: description: orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} duration: 1218.6682364940643 failure_reason: '"1718679275.9375453 mgr.smithi123.olpkyi (mgr.14236) 1 : cluster [ERR] Failed to load ceph-mgr modules: k8sevents" in cluster log' flavor: default owner: scheduled_teuthology@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=cdd56726f7434f7b917174f03ae409db status: fail success: false 2024-06-18T03:04:42.742 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-06-18T03:04:42.825 INFO:teuthology.run:FAIL